How Many Skills Are We Measuring?

Ability Structure Review: Hierarchical vs Multidimensional Models

1 Why this review matters

This page answers one core design question for the screener: should we treat student performance as one hierarchical ability with local probe effects, or as a set of distinct skill dimensions?

The outcome drives how we score and interpret results in downstream reporting.

2 Models compared

Model Interpretation When it is the better fit
M1_1d_testlets One main underlying ability, plus testlet effects for item clusters. Best if students mostly differ on one common trait with local probe structure.
M3_correlated_mirt Multiple skill dimensions (collapsed map), allowed to correlate. Best if students have stable, distinct subskill profiles that can be measured precisely.

3 How we decide

  1. Primary fit criterion (BIC): lower is better after penalising unnecessary complexity.
  2. Residual dependence (Q3): lower residual dependence indicates better structural capture.
  3. Bootstrap robustness: clustered school bootstrap checks whether model preference is stable under resampling.
  4. Profile usability gate (secondary): factor determinacy and SE checks indicate whether profile reporting is operationally usable.

4 Run context

  • Run id: foundation_fulln_boot4_prod_20260209
  • Source path: ../../analysis/structure_model_comparison_v4/outputs/runs/foundation_fulln_boot4_prod_20260209
status converged n
ok TRUE 8

5 Results by unit (Term x slice)

5.1 BIC winners

unit_id slice_id term slice_type scope cohort bic_winner bic_value best_profile_teacher_pass best_profile_internal_pass
main__t1_operational t1_operational 1 operational main NA M1_1d_testlets 166666.28 FALSE FALSE
main__t1_stable_school t1_stable_school 1 stable_school main NA M1_1d_testlets 79592.71 FALSE FALSE
main__t4_operational t4_operational 4 operational main NA M1_1d_testlets 64532.27 FALSE FALSE
main__t4_stable_school t4_stable_school 4 stable_school main NA M1_1d_testlets 62973.63 FALSE FALSE
bic_winner n_units
M1_1d_testlets 4

5.2 Residual dependence (Q3)

unit_id slice_id term slice_type scope cohort model_id mean_abs_q3 max_abs_q3 prop_abs_q3_gt_thresh mean_abs_q3_within_testlet mean_abs_q3_between_testlet
main__t1_operational t1_operational 1 operational main NA M1_1d_testlets 0.0919729 0.9990380 0.1232256 0.1279996 0.0847608
main__t1_operational t1_operational 1 operational main NA M3_correlated_mirt 0.0951650 0.9998940 0.1283517 0.1279433 0.0886031
main__t1_stable_school t1_stable_school 1 stable_school main NA M1_1d_testlets 0.1049479 1.0000000 0.1444590 0.1481031 0.0969043
main__t1_stable_school t1_stable_school 1 stable_school main NA M3_correlated_mirt 0.1064414 1.0000000 0.1467255 0.1477599 0.0987403
main__t4_operational t4_operational 4 operational main NA M1_1d_testlets 0.0996669 0.9940809 0.1237276 0.1369858 0.0925433
main__t4_operational t4_operational 4 operational main NA M3_correlated_mirt 0.1054329 0.9999847 0.1405018 0.1477270 0.0973597
main__t4_stable_school t4_stable_school 4 stable_school main NA M1_1d_testlets 0.1006929 0.9938343 0.1265950 0.1387477 0.0934289
main__t4_stable_school t4_stable_school 4 stable_school main NA M3_correlated_mirt 0.1074439 0.9999924 0.1482437 0.1497796 0.0993628

5.3 Bootstrap robustness (M3 - M1)

Positive values indicate M1 has lower BIC than M3.

slice_id n_reps median_delta_BIC_M3_minus_M1 p10_delta_BIC_M3_minus_M1 p90_delta_BIC_M3_minus_M1 median_delta_det_M3_minus_M1 p10_delta_det_M3_minus_M1 p90_delta_det_M3_minus_M1
t1_operational 30 2951.016 2726.830 3379.751 NA NA NA
t1_stable_school 30 1308.024 1130.808 1524.877 NA NA NA
t4_operational 30 1635.014 1194.773 1743.589 NA NA NA
t4_stable_school 30 1339.944 1190.178 1585.222 NA NA NA
all_slices 120 1536.303 1183.367 3062.105 NA NA NA
Slice Bootstrap rows Usable delta-BIC rows
t1_operational 30 30
t1_stable_school 30 28
t4_operational 30 17
t4_stable_school 30 14

5.4 Profile usability gate

factor_id factor_label factor_role n_students mean_score sd_score mean_se median_se determinacy prop_se_le_teacher prop_se_le_internal teacher_profile_pass internal_profile_pass unit_id model_id slice_id term slice_type scope cohort
G theta general_theta 2514 -0.0022936 0.9375805 0.4852329 0.4832337 0.7838321 0.0175020 0.0473349 NA NA main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S1 BNL0-20 profile 2514 -0.0003807 0.7526043 0.6278531 0.6208052 0.5835345 0.0039777 0.0091488 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S2 DMT10 profile 2514 -0.0035311 0.5389092 0.8254036 0.8102191 0.2885182 0.0003978 0.0031822 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S3 DMT5 profile 2514 -0.0171522 0.6008561 0.8210611 0.8067347 0.3380350 0.0031822 0.0059666 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S4 MC0-20 profile 2514 -0.0096234 0.6525557 0.7340659 0.7360587 0.4325107 0.0031822 0.0063644 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S5 MNA0-20 profile 2514 -0.0027995 0.4043251 0.8626088 0.8742989 0.1744390 0.0011933 0.0039777 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S6 MNC0-20 profile 2514 -0.0141350 0.4846818 0.8704220 0.8785427 0.2301144 0.0027844 0.0059666 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S7 MQ1-10 profile 2514 -0.0087007 0.5132502 0.8702760 0.8820801 0.2505774 0.0047733 0.0075577 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S8 MQ1-20 profile 2514 -0.0001555 0.5392210 0.8542128 0.8591561 0.2756766 0.0015911 0.0027844 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S9 UNLC0-20 profile 2514 -0.0048651 0.5695132 0.7766644 0.7701894 0.3320319 0.0031822 0.0079554 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
S10 UNLNC0-20 profile 2514 -0.0045607 0.5964592 0.7557009 0.6989012 0.3623812 0.0063644 0.0111376 FALSE FALSE main__t1_operational M1_1d_testlets t1_operational 1 operational main NA
F1 DMT10 profile 2514 0.0015320 0.8073703 0.7244806 0.7327162 0.5427607 0.0007955 0.0023866 FALSE FALSE main__t1_operational M3_correlated_mirt t1_operational 1 operational main NA
F2 DMT5 profile 2514 -0.0037046 0.9177859 0.7083367 0.6904556 0.6171578 0.0019889 0.0039777 FALSE FALSE main__t1_operational M3_correlated_mirt t1_operational 1 operational main NA
F3 MQ1-10 profile 2514 -0.0022881 0.7466140 0.8032839 0.8227615 0.4562255 0.0047733 0.0071599 FALSE FALSE main__t1_operational M3_correlated_mirt t1_operational 1 operational main NA
F4 MQ1-20 profile 2514 -0.0042983 0.7325760 0.7896005 0.8192936 0.4525188 0.0007955 0.0035800 FALSE FALSE main__t1_operational M3_correlated_mirt t1_operational 1 operational main NA
F5 NL profile 2514 -0.0033449 0.9662964 0.5028992 0.4974806 0.7844818 0.0043755 0.0115354 FALSE FALSE main__t1_operational M3_correlated_mirt t1_operational 1 operational main NA
F6 SYM_MAG profile 2514 -0.0056075 1.0858509 0.5370743 0.5450725 0.7970843 0.0218775 0.0465394 FALSE FALSE main__t1_operational M3_correlated_mirt t1_operational 1 operational main NA
G theta general_theta 1186 -0.0046138 0.9156667 0.4740175 0.4712176 0.7835355 0.0185497 0.0573356 NA NA main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S1 BNL0-20 profile 1186 -0.0007283 0.7426784 0.6326830 0.6240977 0.5736227 0.0042159 0.0059022 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S2 DMT10 profile 1186 -0.0090578 0.5361078 0.8150418 0.7883479 0.2912247 0.0000000 0.0050590 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S3 DMT5 profile 1186 -0.0146353 0.5733894 0.8268832 0.8231037 0.3148238 0.0033727 0.0067454 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S4 MC0-20 profile 1186 -0.0033654 0.6605251 0.7285331 0.7371220 0.4422232 0.0050590 0.0101180 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S5 MNA0-20 profile 1186 -0.0123532 0.4225218 0.8526970 0.8645486 0.1905067 0.0016863 0.0067454 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S6 MNC0-20 profile 1186 -0.0162130 0.4582293 0.8797758 0.8856165 0.2076351 0.0033727 0.0059022 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S7 MQ1-10 profile 1186 -0.0017042 0.4976512 0.8754485 0.8938772 0.2371139 0.0067454 0.0092749 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S8 MQ1-20 profile 1186 -0.0031609 0.5540096 0.8426898 0.8453828 0.2922090 0.0025295 0.0042159 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S9 UNLC0-20 profile 1186 -0.0018074 0.5830908 0.7713142 0.7219905 0.3452817 0.0042159 0.0075885 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
S10 UNLNC0-20 profile 1186 -0.0119506 0.5932439 0.7545618 0.7195874 0.3609236 0.0050590 0.0109612 FALSE FALSE main__t1_stable_school M1_1d_testlets t1_stable_school 1 stable_school main NA
F1 DMT10 profile 1186 0.0049724 0.7867555 0.7182720 0.6944760 0.5345366 0.0016863 0.0025295 FALSE FALSE main__t1_stable_school M3_correlated_mirt t1_stable_school 1 stable_school main NA
F2 DMT5 profile 1186 0.0051719 0.9049942 0.7064030 0.7093917 0.6128093 0.0008432 0.0042159 FALSE FALSE main__t1_stable_school M3_correlated_mirt t1_stable_school 1 stable_school main NA
F3 MQ1-10 profile 1186 0.0089380 0.7205880 0.8116234 0.8378202 0.4336150 0.0042159 0.0067454 FALSE FALSE main__t1_stable_school M3_correlated_mirt t1_stable_school 1 stable_school main NA
F4 MQ1-20 profile 1186 -0.0029835 0.7196148 0.7900881 0.8041382 0.4436272 0.0016863 0.0050590 FALSE FALSE main__t1_stable_school M3_correlated_mirt t1_stable_school 1 stable_school main NA
F5 NL profile 1186 0.0053365 0.9606860 0.4934980 0.4860451 0.7891116 0.0042159 0.0075885 FALSE FALSE main__t1_stable_school M3_correlated_mirt t1_stable_school 1 stable_school main NA
F6 SYM_MAG profile 1186 0.0048106 1.0706982 0.5298785 0.5356894 0.7973879 0.0160202 0.0446880 FALSE FALSE main__t1_stable_school M3_correlated_mirt t1_stable_school 1 stable_school main NA
G theta general_theta 1098 -0.0006674 0.8611016 0.5483594 0.5453381 0.7052973 0.0163934 0.0264117 NA NA main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S1 BNL0-20 profile 1098 -0.0095298 0.7950847 0.6025998 0.5913511 0.6291363 0.0081967 0.0100182 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S2 DMT10V2 profile 1098 -0.0036353 0.4928012 0.8499703 0.8478582 0.2440505 0.0081967 0.0100182 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S3 DMT5V2 profile 1098 -0.0100873 0.5439342 0.8638590 0.8680178 0.2762527 0.0063752 0.0100182 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S4 MC0-20 profile 1098 -0.0005598 0.6422155 0.7800545 0.7969153 0.3960381 0.0100182 0.0163934 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S5 MNA0-20 profile 1098 -0.0049435 0.4221478 0.8793130 0.9039351 0.1818827 0.0036430 0.0063752 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S6 MNC0-20 profile 1098 0.0022362 0.4558784 0.8915199 0.9138304 0.2024404 0.0054645 0.0118397 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S7 MQ1-10 profile 1098 -0.0140572 0.4336013 0.9063801 0.9243157 0.1817502 0.0063752 0.0118397 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S8 MQ1-20 profile 1098 -0.0029685 0.4907892 0.8795569 0.9101611 0.2310898 0.0063752 0.0109290 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S9 UNLC0-20 profile 1098 -0.0083354 0.6188924 0.7429593 0.7394039 0.3838861 0.0081967 0.0163934 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
S10 UNLNC0-20 profile 1098 0.0021982 0.6297879 0.7189179 0.5921388 0.4043522 0.0081967 0.0182149 FALSE FALSE main__t4_operational M1_1d_testlets t4_operational 4 operational main NA
F1 DMT10V2 profile 1098 0.0108331 0.7107309 0.7871533 0.7820039 0.4396429 0.0018215 0.0018215 FALSE FALSE main__t4_operational M3_correlated_mirt t4_operational 4 operational main NA
F2 DMT5V2 profile 1098 0.0063024 0.7684117 0.7747409 0.7730618 0.4882169 0.0054645 0.0091075 FALSE FALSE main__t4_operational M3_correlated_mirt t4_operational 4 operational main NA
F3 MQ1-10 profile 1098 0.0125904 0.5663417 0.8713015 0.8874534 0.2936683 0.0036430 0.0045537 FALSE FALSE main__t4_operational M3_correlated_mirt t4_operational 4 operational main NA
F4 MQ1-20 profile 1098 0.0013486 0.6345681 0.8599465 0.8920131 0.3461107 0.0009107 0.0036430 FALSE FALSE main__t4_operational M3_correlated_mirt t4_operational 4 operational main NA
F5 NL profile 1098 0.0058129 0.8760666 0.4848274 0.4722885 0.7628263 0.0054645 0.0091075 FALSE FALSE main__t4_operational M3_correlated_mirt t4_operational 4 operational main NA
F6 SYM_MAG profile 1098 0.0093365 0.9667365 0.6252915 0.6448492 0.6969497 0.0182149 0.0346084 FALSE FALSE main__t4_operational M3_correlated_mirt t4_operational 4 operational main NA
G theta general_theta 1076 -0.0019767 0.8575133 0.5520001 0.5492812 0.7008090 0.0157993 0.0269517 NA NA main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S1 BNL0-20 profile 1076 -0.0079166 0.7935180 0.5992228 0.5886780 0.6307754 0.0083643 0.0120818 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S2 DMT10V2 profile 1076 -0.0046678 0.4937920 0.8477333 0.8398419 0.2457432 0.0065056 0.0092937 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S3 DMT5V2 profile 1076 -0.0110831 0.5411544 0.8674879 0.8695187 0.2726624 0.0055762 0.0102230 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S4 MC0-20 profile 1076 0.0002918 0.6411765 0.7839590 0.7999654 0.3930991 0.0111524 0.0148699 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S5 MNA0-20 profile 1076 -0.0081215 0.4186639 0.8778390 0.9028148 0.1799228 0.0037175 0.0074349 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S6 MNC0-20 profile 1076 0.0033193 0.4393169 0.8960394 0.9169248 0.1893288 0.0074349 0.0157993 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S7 MQ1-10 profile 1076 -0.0129926 0.4285020 0.9083253 0.9272600 0.1777166 0.0065056 0.0120818 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S8 MQ1-20 profile 1076 -0.0034420 0.4913753 0.8779322 0.9072424 0.2322691 0.0065056 0.0111524 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S9 UNLC0-20 profile 1076 -0.0083101 0.6241668 0.7396870 0.7119910 0.3898760 0.0074349 0.0139405 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
S10 UNLNC0-20 profile 1076 0.0015230 0.6276054 0.7223173 0.6004953 0.4004615 0.0083643 0.0185874 FALSE FALSE main__t4_stable_school M1_1d_testlets t4_stable_school 4 stable_school main NA
F1 DMT10V2 profile 1076 0.0127832 0.7177660 0.7791455 0.7688488 0.4491633 0.0037175 0.0065056 FALSE FALSE main__t4_stable_school M3_correlated_mirt t4_stable_school 4 stable_school main NA
F2 DMT5V2 profile 1076 0.0154161 0.7314535 0.8005193 0.7847449 0.4471832 0.0055762 0.0074349 FALSE FALSE main__t4_stable_school M3_correlated_mirt t4_stable_school 4 stable_school main NA
F3 MQ1-10 profile 1076 0.0140685 0.5438256 0.8813171 0.8949087 0.2722329 0.0055762 0.0074349 FALSE FALSE main__t4_stable_school M3_correlated_mirt t4_stable_school 4 stable_school main NA
F4 MQ1-20 profile 1076 0.0116678 0.6312111 0.8588685 0.8845747 0.3439805 0.0018587 0.0037175 FALSE FALSE main__t4_stable_school M3_correlated_mirt t4_stable_school 4 stable_school main NA
F5 NL profile 1076 0.0087255 0.8759182 0.4874133 0.4749904 0.7607887 0.0065056 0.0092937 FALSE FALSE main__t4_stable_school M3_correlated_mirt t4_stable_school 4 stable_school main NA
F6 SYM_MAG profile 1076 0.0196471 0.9503663 0.6329590 0.6525982 0.6847685 0.0111524 0.0353160 FALSE FALSE main__t4_stable_school M3_correlated_mirt t4_stable_school 4 stable_school main NA

6 Decision and implication

**Decision:** M1_1d_testlets is preferred in all evaluated units.

**Implication:** treat the accuracy-side construct as primarily hierarchical (one dominant ability) with testlet structure, rather than as a stable multidimensional profile framework for operational reporting.

7 Known caveats

  • Some bootstrap replicates can have unavailable fit statistics for M3; the table above reports usable replicate counts explicitly.
  • Bootstrap profile-precision deltas may be unavailable in some runs; structural fit and bootstrap delta-BIC are treated as primary.