pemsley/coot/model-building

Key Lessons from Chain A Refinement Session

Workflow Checklist - Follow This for Model-Building and Refinement

Best practices for fixing any issue:

1. ☐ ALWAYS recentre first — every time you move to a new residue or problem, call this BEFORE anything else:

coot.set_go_to_atom_chain_residue_atom_name(chain, resno, "CA") or coot.set_rotation_centre(x, y, z) for a map position. The user is watching the screen. If you switch targets without recentring, they will be looking at the wrong place while you work elsewhere. This is confusing and unacceptable.

1. ☐ Check current metrics (Rama/correlation/overlaps)
2. ☐ Make checkpoint if trying something experimental
3. ☐ Apply fix
4. ☐ Re-check metrics to confirm improvement
5. ☐ If worse, restore checkpoint

Jiggle Fitting a Whole Chain

When to use jiggle fitting

Use fit_chain_to_map_by_random_jiggle_and_blur() when a whole chain needs to be placed or re-placed in the map — before any residue-level refinement. Note, however, that the fit is only local. If the whole chain is (say) 10 or more Angstroms away from the real position, then fit_chain_to_map_by_random_jiggle_and_blur() will not work. The per-residue density correlation graph is the key diagnostic: if the mean correlation across the whole chain is near zero with high variance (many residues below 0.4, many negative), the chain is in the wrong place and jiggle fitting is the right first step.

Do not attempt residue-level fixes (rotamers, pepflips, refinement) until the chain is correctly placed. Those tools cannot recover a globally misplaced chain.

Diagnosing misplacement from the correlation graph

Before jiggling, always get a fresh per-residue correlation to use as a true baseline:

stats = coot.map_to_model_correlation_stats_per_residue_range_py(
    imol, chain_id, imol_map, 1, 0)
corrs = [entry[1][1] for entry in stats[0]]
mean_corr = sum(corrs) / len(corrs)
n_below = sum(1 for c in corrs if c < 0.4)
print(f"Mean correlation: {mean_corr:.3f}, residues below 0.4: {n_below}/{len(corrs)}")

Interpretation:

• Mean > 0.7, few residues below 0.4 → chain is well-placed, proceed to residue-level work
• Mean 0.4–0.7, moderate number below 0.4 → chain placed but needs refinement
• Mean near zero, high variance, many negative → chain is in the wrong place, jiggle fit first

Why negative correlations occur in cryo-EM maps: cryo-EM maps are Fourier-based (no F000 term), so the mean density is zero. Negative density exists between atomic features as a mathematical consequence of the reconstruction — not a physical reality. A misplaced chain within the molecular envelope samples this real structured signal (both positive peaks and negative troughs) from the wrong part of the map, producing scattered correlations around zero. This is distinct from sitting in solvent, which gives low-magnitude, low-variance correlations.

Running the jiggle fit

Set the refinement map first, then use the blur variant for cryo-EM:

coot.set_imol_refinement_map(imol_map)
result = coot.fit_chain_to_map_by_random_jiggle_and_blur(
    imol,          # model molecule
    chain_id,      # e.g. "N"
    n_trials,      # 200-500; more trials = better chance of finding global optimum
    jiggle_scale,  # 1.0-2.0; scale of random perturbations
    blur_factor    # 100-200; Fourier filtering makes the map smoother, helping
                   # the fit escape local minima and find the correct placement
)
print(f"Jiggle fit score: {result:.3f}")

Return values: the function returns a score (higher = better fit) or a large negative number (e.g. −100, −999) when it cannot improve on the current placement. These large negative returns are not errors — they mean the current position is already a local optimum, or the function had difficulty scoring. Do not treat them as failures without checking the actual correlation afterwards.

Blur factor guidance: the blur smooths the map by a something like a Fourier filter, effectively low-pass filtering it. A higher blur factor produces a smoother map with broader features, which helps the chain find the correct region before fine-grained fitting. For cryo-EM maps, blur factors of 100–200 are typical starting points.

Confirming the result

Always re-run the per-residue correlation after jiggling and compare to the baseline:

stats_after = coot.map_to_model_correlation_stats_per_residue_range_py(
    imol, chain_id, imol_map, 1, 0)
corrs_after = [entry[1][1] for entry in stats_after[0]]
mean_after = sum(corrs_after) / len(corrs_after)
n_below_after = sum(1 for c in corrs_after if c < 0.4)
print(f"AFTER  — mean: {mean_after:.3f}, below 0.4: {n_below_after}")

A successful jiggle fit for a correctly placed chain should bring the mean correlation above 0.6, with most residues above 0.7. Remaining poor-fitting residues in loop regions are expected and addressed by subsequent residue-level refinement.

Visualising the before/after result

Use the coot-inline-graphs skill to render a before/after bar chart with secondary structure overlays. Fetch secondary structure from PDBe (/api/pdb/entry/secondary_structure/{pdb_id}) rather than relying solely on Coot's header secondary structure detection, which may miss strands — particularly single- residue assignments and short strands in nanobody/VHH-type folds.

Complete workflow summary

# 1. Get fresh baseline correlation (after any manual moves, before jiggle)
stats_before = coot.map_to_model_correlation_stats_per_residue_range_py(
    imol, chain_id, imol_map, 1, 0)
corrs_before = [e[1][1] for e in stats_before[0]]
print(f"BEFORE — mean: {sum(corrs_before)/len(corrs_before):.3f}")
# 2. Set refinement map and jiggle fit
coot.set_imol_refinement_map(imol_map)
result = coot.fit_chain_to_map_by_random_jiggle_and_blur(imol, chain_id, 500, 1.0, 200.0)
print(f"Jiggle score: {result:.3f}")  # large negative = ignore, check correlation anyway
# 3. Get after correlation and compare
stats_after = coot.map_to_model_correlation_stats_per_residue_range_py(
    imol, chain_id, imol_map, 1, 0)
corrs_after = [e[1][1] for e in stats_after[0]]
print(f"AFTER  — mean: {sum(corrs_after)/len(corrs_after):.3f}")
# 4. Only proceed to residue-level work once mean > 0.6

Refinement Best Practices

1. Make Checkpoints Before Model Changes

CRITICAL: Always create a checkpoint before making significant model changes.

Use make_backup_checkpoint() before any operation that might need to be reverted:

• Adding/deleting residues
• Adding ligands or waters
• Major refinement operations
• Any experimental model building

# Create a named checkpoint before risky operation
checkpoint_idx = coot.make_backup_checkpoint(0, "before adding OXT")
# Try the operation
coot.add_OXT_to_residue(0, "A", 93, "")
result = coot.refine_residues_py(0, [["A", 93, ""]])
# Check if it worked - if not, restore
if result_is_bad:
    coot.restore_to_backup_checkpoint(0, checkpoint_idx)

Why checkpoints are better than undo:

• apply_undo() only steps back one operation at a time
• Checkpoints let you jump back to a specific point
• Named checkpoints are self-documenting
• Multiple checkpoints allow comparing different approaches

# Compare two different approaches
checkpoint_before = coot.make_backup_checkpoint(0, "original state")
# Try approach 1
coot.auto_fit_best_rotamer(0, "A", 42, "", "", 1, 1, 0.01)
results = coot.refine_residues_py(0, [["A", 41, ""], ["A", 42, ""], ["A", 43, ""]])
score_approach1 = check_correlation(0, "A", 42)
checkpoint_approach1 = coot.make_backup_checkpoint(0, "after approach 1")
# Restore and try approach 2
coot.restore_to_backup_checkpoint(0, checkpoint_before)
coot.pepflip(0, "A", 42, "", "")
results = coot.refine_residues_py(0, [["A", 41, ""], ["A", 42, ""], ["A", 43, ""]])
score_approach2 = check_correlation(0, "A", 42)
# Keep the better result
if score_approach1 > score_approach2:
    coot.restore_to_backup_checkpoint(0, checkpoint_approach1)

MANDATORY: Complete Validation Workflow

CRITICAL: You MUST check ALL validation metrics before AND after EVERY fix.

Fixing only one problem (e.g., Ramachandran) while leaving others (rotamer, density fit) is a FAILED fix. A residue is only "fixed" when ALL metrics are acceptable.

Before fixing ANY residue:

1. ALWAYS center on it: coot.set_go_to_atom_chain_residue_atom_name(chain, resno, "CA")
2. ALWAYS check ALL of these:

• Ramachandran probability (from all_molecule_ramachandran_score_py)
• Rotamer score (from rotamer_graphs_py)
• Density correlation - both all-atom and side-chain (from map_to_model_correlation_stats_per_residue_range_py)
• Atom overlaps involving this residue (from molecule_atom_overlaps_py)

After fixing ANY residue:

1. ALWAYS re-check ALL the same metrics, Sometime residues/issues are just

not fixable (that's what makes refinement and model-building tricky).

1. Try to make the following true (but knowing that it might not actually be possible)

• Ramachandran probability > 0.02 (preferably > 0.1)
• Rotamer score > 1.0% (preferably > 5%)
• Density correlation > 0.7 (all-atom and side-chain, preferably > 0.8)
• No severe clashes (< 2.0 Å cubed overlap volume)

If ANY metric is still bad after your first fix:

1. MUST try additional fixes:

• Bad rotamer → auto_fit_best_rotamer(), and try experiment with following that up

with refine_residues_py() for that residue and its upstream and downstream neighbours (if any).

• Poor density fit → try alternative rotamers, check for missing atoms
• Persistent clashes → refine with the addition of spatial neighbours using

residues_near_residue_py()

1. NEVER declare a residue "fixed" based on only one metric improving
2. ALWAYS re-validate after each additional fix

Acceptable Reasons to Stop (without perfect metrics):

• You've tried multiple approaches and documented them
• The best achievable metrics are still recorded
• You've created checkpoints to compare approaches
• You explain why the residue remains problematic (e.g., poor density, crystal contact)

Example of CORRECT Workflow

# 1. ALWAYS center on problem residue first
coot.set_go_to_atom_chain_residue_atom_name("A", 41, "CA")
# 2. Get ALL metrics BEFORE
rama_data = [r for r in coot.all_molecule_ramachandran_score_py(0)[5:]
             if r[1] == ['A', 41, '']][0]
rama_prob_before = rama_data[2]
rotamer_data = [r for r in coot.rotamer_graphs_py(0)
                if r[0] == 'A' and r[1] == 41][0]
rotamer_score_before = rotamer_data[3]
corr_data = [s for s in coot.map_to_model_correlation_stats_per_residue_range_py(0, "A", 1, 1, 0)[0]
             if s[0][1] == 41][0]
correlation_before = corr_data[1][1]
overlaps_before = [o for o in coot.molecule_atom_overlaps_py(0, 30)
                   if (o['atom-1-spec'][1:3] == ['A', 41] or
                       o['atom-2-spec'][1:3] == ['A', 41])]
print(f"BEFORE: Rama={rama_prob_before:.4f}, Rotamer={rotamer_score_before:.2f}%, Corr={correlation_before:.3f}, Clashes={len(overlaps_before)}")
# 3. Apply first fix (e.g., pepflip for backbone)
coot.pepflip(0, "A", 41, "", "")
coot.refine_residues_py(0, [["A", 40, ""], ["A", 41, ""], ["A", 42, ""], ["A", 43, ""]])
# 4. Check ALL metrics AFTER first fix
rama_prob_after = [r for r in coot.all_molecule_ramachandran_score_py(0)[5:] 
                   if r[1] == ['A', 41, '']][0][2]
rotamer_score_after = [r for r in coot.rotamer_graphs_py(0) 
                       if r[0] == 'A' and r[1] == 41][0][3]
correlation_after = [s for s in coot.map_to_model_correlation_stats_per_residue_range_py(0, "A", 1, 1, 0)[0] 
                     if s[0][1] == 41][0][1][1]
print(f"AFTER:  Rama={rama_prob_after:.4f}, Rotamer={rotamer_score_after:.2f}%, Corr={correlation_after:.3f}")