Replay software versions: {"python": "3.12.3 (tags/v3.12.3:f6650f9, Apr  9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)]", "rdkit": "2026.03.5", "sklearn": "1.9.0", "pandas": "3.0.5", "numpy": "2.4.6", "scipy": "1.17.0", "matplotlib": "3.8.4", "requests": "2.32.5"}
Disposition counts: {'included_representative': 5395, 'missing_label': 2021, 'consistent_duplicate_collapsed': 161, 'multiple_carbon_fragments': 144, 'no_carbon_parent': 64, 'unsupported_parent_element': 20, 'conflicting_identity_labels': 20, 'invalid_smiles': 6}
Unique identity conflicts: 9
                n  active  scaffolds  inactive  prevalence  fraction
split                                                               
test          945     168        505       777    0.177778  0.175162
calibration   942     168        505       774    0.178344  0.174606
train        3508     518        602      2990    0.147662  0.650232
Largest scaffold groups: {('__ACYCLIC__', 'train'): 1313, ('c1ccccc1', 'train'): 1119, ('c1ccncc1', 'train'): 65, ('c1ccc(Cc2ccccc2)cc1', 'calibration'): 59, ('c1ccc2ccccc2c1', 'test'): 52, ('C1CCCCC1', 'calibration'): 46}
Split checksum: fc61b12a2de25923c149da913a43728b6fbed077d73b7a77d4bee80b9c9ceaa8
    C  mean_AP    sd_AP  mean_ROC_AUC  mean_Brier
 0.01 0.308299 0.138694      0.609193    0.148204
 0.10 0.413061 0.104089      0.738260    0.132257
 1.00 0.422935 0.039685      0.750330    0.145661
10.00 0.393703 0.024362      0.734770    0.176095
{
  "best_C": 1.0,
  "calibration_intercept": -0.7219269985650858,
  "calibration_slope": 0.4399019024545635,
  "selected_threshold": 0.18303976287593463,
  "calibration_F1_at_selected_threshold": 0.5237113402061856,
  "training_prevalence": 0.1476624857468643,
  "fit_seconds": 3.0436222999705933,
  "model_n_iter": 9,
  "warnings": []
}
Test probabilities and test performance have not yet been computed.
     model            metric  estimate   ci_low  ci_high
prevalence average_precision  0.177778 0.131043 0.228939
prevalence           roc_auc  0.500000 0.500000 0.500000
prevalence             brier  0.147080 0.114147 0.183132
prevalence          log_loss  0.471427 0.389498 0.561115
       raw average_precision  0.515864 0.392912 0.625607
       raw           roc_auc  0.797186 0.733987 0.849069
       raw             brier  0.129641 0.107921 0.153250
       raw          log_loss  0.444370 0.365076 0.530506
calibrated average_precision  0.515864 0.392912 0.625607
calibrated           roc_auc  0.797186 0.733987 0.849069
calibrated             brier  0.115279 0.094040 0.139364
calibrated          log_loss  0.375025 0.318068 0.436636
Paired differences (negative favors calibrated):
           metric                  comparison  estimate    ci_low   ci_high  bootstrap_valid
            brier        calibrated minus raw -0.014363 -0.025423 -0.004135             1000
            brier calibrated minus prevalence -0.031801 -0.050271 -0.014471             1000
         log_loss        calibrated minus raw -0.069345 -0.111368 -0.031462             1000
         log_loss calibrated minus prevalence -0.096401 -0.145803 -0.048535             1000
ECE_10_fixed_bins        calibrated minus raw -0.057207 -0.075132 -0.009939             1000
ECE_10_fixed_bins calibrated minus prevalence -0.006329 -0.033469  0.047250             1000
Operating points:
     model threshold_label      metric   estimate     ci_low    ci_high
       raw             0.5          tn 694.000000 590.950000 819.000000
       raw             0.5          fp  83.000000  60.000000 111.000000
       raw             0.5          fn  86.000000  56.000000 126.025000
       raw             0.5          tp  82.000000  53.000000 115.000000
       raw             0.5   precision   0.496970   0.380922   0.602504
       raw             0.5      recall   0.488095   0.381780   0.589346
       raw             0.5 specificity   0.893179   0.867454   0.918231
calibrated             0.5          tn 753.000000 640.975000 896.025000
calibrated             0.5          fp  24.000000  15.000000  35.000000
calibrated             0.5          fn 128.000000  87.000000 183.000000
calibrated             0.5          tp  40.000000  22.000000  64.000000
calibrated             0.5   precision   0.625000   0.461538   0.753251
calibrated             0.5      recall   0.238095   0.147047   0.346237
calibrated             0.5 specificity   0.969112   0.954954   0.982166
calibrated  development_F1          tn 558.000000 476.950000 652.050000
calibrated  development_F1          fp 219.000000 170.975000 281.025000
calibrated  development_F1          fn  43.000000  26.000000  63.000000
calibrated  development_F1          tp 125.000000  79.975000 181.025000
calibrated  development_F1   precision   0.363372   0.273721   0.452014
calibrated  development_F1      recall   0.744048   0.631907   0.830508
calibrated  development_F1 specificity   0.718147   0.674555   0.759785
Calibration diagnostics: [{'model': 'raw', 'intercept_joint': -0.7754332934154435, 'slope_joint': 0.45247745912163867, 'note': 'Test-set diagnostic only, not used to change predictions; ideal intercept0 slope1.'}, {'model': 'calibrated', 'intercept_joint': -0.032784609271900265, 'slope_joint': 1.0285900864251778, 'note': 'Test-set diagnostic only, not used to change predictions; ideal intercept0 slope1.'}]
Coverage heuristic: {"training_reference_n": 3508, "calibration_similarity_q10": 0.25024271844660195, "descriptor_bounds_quantiles": [0.01, 0.99], "definition": "Supported by heuristic screen iff similarity >= calibration q10 AND all descriptors within training q01-q99.", "limitation": "Heuristic flags, not validated guarantees. Low-coverage predictions remain in all primary metrics.", "ranking_minimums": {"n": 50, "active": 10, "inactive": 10, "scaffolds": 10}}
         scheme   stratum   n  active  n_scaffolds  average_precision  roc_auc    brier  log_loss
     similarity   [0,0.3) 211      19          153           0.548833 0.850055 0.065209  0.244696
     similarity [0.3,0.5) 479      87          288           0.437461 0.768869 0.126967  0.405657
     similarity [0.5,0.7) 227      60          121           0.673835 0.803493 0.147621  0.460568
     similarity   [0.7,1]  28       2           22                NaN      NaN 0.030436  0.139620
coverage_screen supported 773     158          401           0.550160 0.795323 0.127298  0.406066
coverage_screen   flagged 172      10          135           0.506746 0.825309 0.061263  0.235521
Training descriptor envelopes:
      descriptor  training_q01  training_q99  training_min  training_max
molecular_weight      71.15096    800.930790       27.0260    1473.61400
           cLogP      -2.00907      7.717848      -10.1699      18.55122
            TPSA       0.00000    206.035300        0.0000     639.10000
             HBD       0.00000      6.000000        0.0000      21.00000
             HBA       0.00000     13.000000        0.0000      32.00000
 rotatable_bonds       0.00000     21.000000        0.0000      47.00000
      ring_count       0.00000      6.000000        0.0000      12.00000
     heavy_atoms       4.00000     55.000000        2.0000     102.00000
Nearest-training similarity quantiles: {0.0: 0.14285714285714285, 0.1: 0.24893617021276596, 0.25: 0.3090909090909091, 0.5: 0.4, 0.75: 0.5, 0.9: 0.5944838455476754, 1.0: 1.0}
Exact fingerprint similarities of 1: 6
Identical-fingerprint pairs:
       compound_id                                     identity                               nearest_training_identity  y  nearest_training_label  p_calibrated
M_c46aab0487d992b1          Cc1ccc(S(=O)(=O)NC(=O)NC2CCCCC2)cc1                   Cc1ccc(S(=O)(=O)NC(=O)NC2CCCCCCC2)cc1  0                       0      0.176883
M_bd8014e536457362                                  O=NN1CCCCC1                                           O=NN1CCCCCCC1  0                       0      0.043491
M_28ed7f4abc3f3609                                   O=C1CCCCC1                                        O=C1CCCCCCCCCCC1  0                       0      0.167257
M_daa04a326d407f10                                     C1CCNCC1                                              C1CCCNCCC1  0                       0      0.063239
M_21454a3efb52eb34            C[Si]1(C)O[Si](C)(C)O[Si](C)(C)O1 C[Si]1(C)O[Si](C)(C)O[Si](C)(C)O[Si](C)(C)O[Si](C)(C)O1  0                       0      0.062952
M_55a550b608c5667a C[Si]1(C)O[Si](C)(C)O[Si](C)(C)O[Si](C)(C)O1 C[Si]1(C)O[Si](C)(C)O[Si](C)(C)O[Si](C)(C)O[Si](C)(C)O1  0                       0      0.062952
Sensitivity:
                               analysis      model   n  active  average_precision  roc_auc    brier
                              full_test prevalence 945     168           0.177778 0.500000 0.147080
                              full_test        raw 945     168           0.515864 0.797186 0.129641
                              full_test calibrated 945     168           0.515864 0.797186 0.115279
exclude_identical_training_fingerprints prevalence 939     168           0.178914 0.500000 0.147880
exclude_identical_training_fingerprints        raw 939     168           0.516194 0.796581 0.130435
exclude_identical_training_fingerprints calibrated 939     168           0.516194 0.796581 0.115938
Calibration bin counts:  bin   n  active  n_scaffolds  mean_probability  observed_fraction
   0 377      22          242          0.052507           0.058355
   1 255      25          169          0.142613           0.098039
   2 131      32           85          0.240435           0.244275
   3  65      24           47          0.347747           0.369231
   4  53      25           41          0.445593           0.471698
   5  35      20           28          0.546418           0.571429
   6  10       3            9          0.649689           0.300000
   7   7       6            6          0.745875           0.857143
   8   6       5            6          0.855090           0.833333
   9   6       6            4          0.942935           1.000000
Coverage uncertainty:
         scheme   stratum            metric  estimate   ci_low  ci_high
     similarity   [0,0.3) average_precision  0.548833 0.190379 0.785355
     similarity   [0,0.3)           roc_auc  0.850055 0.728389 0.944176
     similarity   [0,0.3)             brier  0.065209 0.046186 0.088809
     similarity [0.3,0.5) average_precision  0.437461 0.280427 0.581685
     similarity [0.3,0.5)           roc_auc  0.768869 0.666769 0.841955
     similarity [0.3,0.5)             brier  0.126967 0.098983 0.158805
     similarity [0.5,0.7) average_precision  0.673835 0.524038 0.799526
     similarity [0.5,0.7)           roc_auc  0.803493 0.721761 0.873976
     similarity [0.5,0.7)             brier  0.147621 0.112318 0.190424
     similarity   [0.7,1] average_precision       NaN      NaN      NaN
     similarity   [0.7,1]           roc_auc       NaN      NaN      NaN
     similarity   [0.7,1]             brier  0.030436 0.008280 0.059193
coverage_screen supported average_precision  0.550160 0.421988 0.657621
coverage_screen supported           roc_auc  0.795323 0.725173 0.851684
coverage_screen supported             brier  0.127298 0.101653 0.154264
coverage_screen   flagged average_precision  0.506746 0.167989 0.807272
coverage_screen   flagged           roc_auc  0.825309 0.619952 0.974953
coverage_screen   flagged             brier  0.061263 0.041168 0.084987
       compound_id nearest_training_compound_id  unfolded_feature_presence_equal  unfolded_feature_counts_equal  test_heavy_atoms  training_heavy_atoms
M_c46aab0487d992b1           M_f0aee7a83e19d05e                             True                          False                20                    22
M_bd8014e536457362           M_a25310c7027dba6a                             True                          False                 8                    10
M_28ed7f4abc3f3609           M_208c48ab7148e1f9                             True                          False                 7                    13
M_daa04a326d407f10           M_d15476494a0d85af                             True                          False                 6                     8
M_21454a3efb52eb34           M_f85beaa7d4cd4cbd                             True                          False                12                    20
M_55a550b608c5667a           M_f85beaa7d4cd4cbd                             True                          False                16                    20
Exported figures: ['calibration.pdf', 'calibration.png', 'chemical_coverage.pdf', 'chemical_coverage.png', 'discrimination.pdf', 'discrimination.png', 'illustrative_compounds.png']
Traceback (most recent call last):
  File "<cell>", line 8, in <module>
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "tox21_assessment\reproduce_analysis.py", line 579, in <module>
    pd.testing.assert_frame_equal(reference_split,clean,check_dtype=False,rtol=1e-12,atol=1e-12)
  File "D:\down\Lib\site-packages\pandas\util\_decorators.py", line 220, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\down\Lib\site-packages\pandas\_testing\asserters.py", line 1347, in assert_frame_equal
    assert_series_equal(
  File "D:\down\Lib\site-packages\pandas\util\_decorators.py", line 193, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\down\Lib\site-packages\pandas\_testing\asserters.py", line 1121, in assert_series_equal
    _testing.assert_almost_equal(
  File "pandas/_libs/testing.pyx", line 53, in pandas._libs.testing.assert_almost_equal
  File "pandas/_libs/testing.pyx", line 171, in pandas._libs.testing.assert_almost_equal
  File "D:\down\Lib\site-packages\pandas\_testing\asserters.py", line 619, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 9] (column name="error") are different

DataFrame.iloc[:, 9] (column name="error") values are different (100.0 %)
[index]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, ...]
[left]:  [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, ...]
[right]: <ArrowStringArray>
['', '', '', '', '', '', '', '', '', '',
 ...
 '', '', '', '', '', '', '', '', '', '']
Length: 5395, dtype: str
At positional index 0, first diff: nan != 
