Designing a Psychometric Compatibility Model for a Home Improvement Matching Platform
Project Summary
We developed a psychometric Compatibility Model to measure and optimise the fit between homeowners (HO) and service providers (SP) collaborating on U.S. home improvement projects. The model is built into our partner's software platform and uses two separate, validated 53-question assessment tools — one for homeowners and one for service providers - that assess five key traits known to affect the results of home improvement projects.
The development process began with an exhaustive review of more than 500 data sources across 15 U.S. states, encompassing customer reviews, industry reports, academic studies, government data, and consumer feedback platforms. This research identified seven critical themes shaping HO-SP interactions, including transaction complexity, information gaps, hiring practices, and business standards. Trust emerged as the single most important predictor of project success, built on clear communication, reliability, collaboration, and accountability.
These insights directly shaped a five-trait assessment framework. We created scenario-based vignette and self-report items, piloted them with 501 homeowners and 289 service providers across two validation phases, and applied psychometric methods — Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA), Item Response Theory (IRT-GRM), reliability testing, and Structural Equation Modeling (SEM) — to confirm that the final instruments are both reliable and accurately measure the intended constructs. We then created an automated scoring system together with narrative feedback reports and provided consultancy on integrating the assessments into the front-end and back-end of the SaaS platform.
The Challenge
Homeowners across the U.S. frequently lack access to reliable, structured information when hiring service providers for home improvement projects. This information asymmetry exposes them to well-documented risks, including hidden costs, inconsistent service quality, vague or incomplete contracts, unpredictable workmanship, and frequent disputes.
Unlike B2B environments, which benefit from formal vetting processes, legal safeguards, and professional procurement standards, most homeowners rely on inconsistent online reviews, word-of-mouth referrals, or rushed decisions driven by time pressure and limited experience. As a result, mismatches often occur in communication style, work ethic, values, and conflict-handling approaches. These misalignments erode trust and cause projects to derail.
The core challenge was to replace subjective or superficial matching criteria - such as price, star ratings, or basic availability - with a data-driven, trait-based compatibility system capable of predicting which homeowner-service provider pairings are most likely to succeed.
An extra challenge was to develop a scoring algorithm grounded in relevant data. Because homeowners and service providers differ demographically, we needed to ensure that both groups were measured on equivalent traits. To achieve this, we applied advanced statistical methods - Multi-Group Confirmatory Factor Analysis, Tucker's Congruence Coefficient, and Procrustes Rotation - to test for measurement invariance and factor structure similarity across the two groups. The analyses showed moderate to high congruence, confirming that the assessments measured the same underlying traits equivalently for both HOs and SPs. We also conducted Canonical Correlation Analysis, which produced moderate coefficients.
Project Goals
- Review and synthesise insights from 500+ real-world data sources to identify the psychological traits most critical to homeowner-service provider compatibility.
- Develop a five-trait assesment framework with detailed sub-components.
- Design and refine scenario-based vignette and self-report items that accurately capture these traits for both homeowners and service providers.
- Pilot the assessments with a demographically representative U.S. sample (501 homeowners + 289 service providers) across two validation phases.
- Apply core statistical methods like EFA, CFA, IRT, SEM, and reliability analysis to refine the items and establish the validity and reliability of the instruments.
- Examine trait score differences between homeowners and service providers and conduct Differential Item Functioning (DIF) analysis for fairness.
- Confirm measurement invariance and structural equivalence between the homeowner and service provider versions using Multi-Group CFA and related techniques.
- Design an automated scoring system with narrative feedback reporting.
- Provide strategic consultancy on integrating the compatibility assessments into the front-end and back-end of the SaaS platform.
Our Solution
The initial research phase synthesised data from +500 sources, confirming that homeowners face major information disadvantages and that trust operationalised through communication, reliability, collaboration, and accountability is the decisive factor in project success.
We piloted the assessments on 849 U.S. English-speaking participants recruited through a certified panel provider with strict demographic balancing. Statistical quality-control methods such as Mahalanobis Distance, Longstring Analysis, Shannon Entropy, Autocorrelation, and Variance, were combined into a composite data-quality score. This removed 62 low-quality responses (11%), yielding a final sample of 501 homeowners and 289 service providers.
Demographic profiling showed homeowners were older on average (42 vs. 38 years), with balanced gender representation, while service providers were predominantly male (80%) and reported extensive experience (often 11-100+ projects). Homeowners typically had limited prior project experience and favoured budgets under $50k. Providers handled $15k-$100k projects and operated across small agencies, self-employment, and larger companies.
Exploratory Factor Analysis (Principal Axis Factoring with oblimin rotation) confirmed factor structures and sampling adequacy. Targeted item reductions were applied using weighted psychometric criteria (loadings, uniqueness, item-rest correlations, fit indices, theory, and expert review). All scales demonstrated acceptable to strong internal consistency after refinement (Cronbach's α > .70, typically .72–.89). Model fit indices (RMSEA, SRMR, TLI, CFI) were generally good to excellent, with minor variations by scale and group. Item Response Theory (Graded Response Model) confirmed strong item discrimination and appropriate thresholds for the majority of items, with only a small number flagged for minor refinement
Multi-Group CFA, Tucker's Congruence Coefficient, and Procrustes Rotation confirmed moderate-to-high measurement invariance and factor similarity between the two versions. Canonical Correlation Analysis further supported the convergent validity of the models.
Building on this validated foundation, we then developed a robust psychometric scoring algorithm and created narrative feedback reports. We also delivered a production-ready TypeScript implementation of the scoring system and provided consultancy services to support seamless integration into both the front-end and back-end of the SaaS platform.