CryoSPARC v4.5 Release Notes

CryoSPARC Overview

CryoSPARC (Cryo-EM Single Particle Ab-initio Reconstruction and Classification) is a software package for processing cryo-electron microscopy (cryo-EM) single-particle data, used in research and drug discovery.

As a complete solution for cryo-EM processing, cryoSPARC allows:

• Ultra-fast end-to-end processing of raw cryo-EM data and reconstruction of electron density maps, ready for ingestion into model-building software
• Optimized algorithms and GPU acceleration at all stages, from pre-processing through particle picking, 2D particle classification, 3D ab-initio structure determination, high-resolution refinement, and heterogeneity analysis
• Specialized and unique tools for therapeutically relevant targets, membrane proteins, and continuously flexible structure
• Interactive, visual, and iterative experimentation for even the most comple

CryoSPARC v4.5 Release

The newest update is packed with new features, performance optimizations, and various stability enhancements. Version 4.5 includes

• a new micrograph denoiser
• automatic 2D/3D class selection using a reference
• improvements to Workflows
• orientation diagnostics
• 3D classification and volume group outputs
• Reference Based Motion Correction improvements
• new tools for handling and aligning multiple 3D volume
• many feature updates, fixes, and optimizations!

Download the latest version here & Guide to updating here

New Features

Micrograph Denoiser

Micrograph Denoiser BETA job produces enhanced contrast in micrograph images to aid particle picking and visual inspection.

• Micrograph Denoiser uses a neural network model that can be trained on as few as 100 exposures from a new dataset in just a few minutes on a single GPU. The trained denoiser can then be used on new micrographs from the same or a different dataset. A pre-trained general denoiser model is also available to use without any training required.
• The denoiser model is designed to work across a wide range of defocus and contrast levels.
• Denoised micrographs can be used seamlessly in blob and template picking jobs, and can substantially improve picking results. Denoised micrographs also show by default in all interactive jobs, making visual inspection of micrographs easy.

Reference Based Auto Select 2D & 3D

Automatic selection of 2D classes or 3D classes given a 3D reference is now possible using new Reference Based Auto Select 2D BETA and Reference Based Auto Select 3D BETA jobs.

• These jobs align and compare classes to the reference volume and select classes either based on score thresholds or clustering.
• The jobs can be set up once on a dataset, and can subsequently allow one-click automated processing on new similar datasets using Workflows.
• The jobs also make it easy to sort and select from hundreds of 2D classes or dozens 3D classes in datasets with substantial heterogeneity.

Orientation Diagnostics Reporting Relative Signal

Orientation Diagnostics job now reports Relative Signal — a new diagnostic that can help directly identify underrepresented views that affect map quality and the cFAR score. Relative signal is computed by considering FSCs within toroidal sections of Fourier half-maps to relate 3D signal quality to viewing direction. Two new plots are produced that show where on the viewing sphere relative signal is attenuated, with sample projections and an embedded structure to simplify interpretation.

Rebalance Orientation

New Rebalance Orientations job removes particles from overpopulated viewing direction regions. Rebalancing viewing directions can improve downstream refinements when particles have a highly non-uniform orientation distribution.

Volume Group Outputs

Jobs that produce multiple 3D volumes (such as 3D classification, Heterogeneous Refinement, 3D Variability Display, etc.) now output a “Volumes group” output that contains multiple volumes, and can be connected to a downstream job with a single input connection. This enables seamless handling of arbitrary numbers of 3D volumes and automation using Workflows.

Align 3D Maps

Align 3D Maps has been reimplemented with a new 3D alignment method to increase the robustness and precision of volume-to-volume alignment. In addition, it can now:

• update particle alignments while searching over handedness,
• transform all maps and masks associated with an input volume slot,
• and accept inputs of different box and pixel sizes.

Workflow Improvements

Direct Browser Upload - Upload files from your local computer to the CryoSPARC system directly in the browser. This new ability streamlines workflows involving local visualization and editing of files such as 3D maps or masks. Uploaded files are saved to the selected project directory and can be imported and used in jobs. Read more in the guide.

• Expandable Search bar - The file browser now features an expandable sidebar to search and navigate into project directories. Bookmark commonly-accessed directories, making them available for access at a later time through the sidebar. Read more in the guide.
• Improved ‘Inputs and Parameters’ tab - Jobs now displays additional input group information in a summarized view while retaining an option to view low-level connection details. By default, only custom parameters are shown, making it easier to identify which parameters were set for a particular job.
• Additional Info on Jobs - Additional info tags now display on the job card with useful stats and summary data, for more jobs including Import Volumes, Exposure Curation, Exposure Sets Tool, and Helical Refinement. New quick actions allow quick creation and connection of many more job types in typical workflows.
• Job Sidebar - The details section of the job sidebar now displays a more comprehensive summary of parents, children, and cloned jobs when hovered, including job type, status, summary stats, and custom parameters.

CryoSPARC v4.5 Updates

• Updates to Workflows:
  • New rebuilding system allows retaining annotations and settings when updating an existing Workflow.
  • Workflows can now be applied from anywhere in the application by selecting or creating a project and workspace from within the Workflow apply dialog.
  • Filter controls have been added to allow collapsing/expanding all jobs in sidebar, filtering by flagged jobs, and navigating to a selected job.
  • Fork option has been replaced with a “Create New” option in the edit dialog for simplicity and clarity.
• Improvements to Reference Based Motion Correction BETA:
  • Input particles are now re-centered based on their aligned shifts, at the start of the job. This feature can be switched off. The upsampling factor for .eer movies can now be overridden.
  • The job has been improved to be significantly more stable. An option has been added to prevent crashing if the input movies contain different frame counts (the option is off by default). This option relies on the assumption that the most common frame count is “correct” and will discard movies that don’t match that frame count. The job also no longer terminates if the distances between particles results in a singular covariance matrix, as a pseudo-inversion is now used. The chance of getting NaN values in the highest-frequency column of the FCC plot (due to low signal near the FSC cut-off frequency) has been greatly reduced. The job also no longer crashes if the upstream motion correction skipped the movie’s first frame.
  • The job now outputs movies or particles that were rejected, as separate output groups. Trajectory plots now correctly reflect the micrograph’s aspect ratio. The mask_refine low-level input is now optional (a mask must be supplied, but if the separate static mask connection is used, mask_refine need not be present).
  • It is now possible to prevent the GPU responsible for generating 2D reference projections from also computing trajectory estimates. Using this option reduces VRAM consumption but requires the use of at least 2 GPUs.
• Refinement jobs now produce a cFSC summary plot
  • at every iteration cFSC summar plot are produced
  • enabling anisotropy tracking throughout optimization (this includes homogeneous, non-uniform, helical, and local refinement).
  • A new fast GPU-based implementation of the core cFSC computation adds minimal overhead, requiring less than a second of additional compute time for typical box sizes.
• Several improvements to 3D Classification:
  • In line with 3D Variability Analysis, the ‘Target Resolution’ parameter has been renamed to ‘Filter Resolution’ and must be explicitly set for the job to run. Also, the secondary RMS-based convergence criterion is now turned on by default. This secondary criterion can significantly reduce run times without substantially affecting the final class volumes.
  • The job now includes particle pre-processing and symmetry enforcement, and class re-ordering is now turned off by default to avoid potential confusion.
  • The job tile figure is now revamped to show class ids, relative populations, and a more distinctive difference-from-consensus projection for each class.
• In Downsample Particles
  • Users can now specify the desired approximate pixel size after Fourier-cropping, instead of specifying the Fourier-crop box size directly, if desired.
• The 3D Flex Generator BETA is now colored
  • latent space plot is now colored according to the normalized difference in 3D deformation relative to the origin. Larger values indicate regions of the latent space that encode more substantial flexing of the particle.
• Regroup 3D now allows for
  • pairwise alignment of volumes prior to regrouping
  • alignment of volumes within a superclass after regrouping
  • regrouping of all volumes into one single superclass.

• The Manual Picker interactive job now writes manually picked particle locations to disk every 10 seconds, to minimize data loss in case the job crashes.

  The new and more reliable SSD cache system introduced in v4.4 is now enabled by default. It also now features a configurable distributed locking strategy to use when multiple running jobs access the SSD cache simultaneously; the default strategy is best in most cases but use the master strategy for caches on distributed file systems such as GPFS and BeeGFS where POSIX locks are disabled or unavailable. See guide for details.

• Deleting a project now completely removes the project folder on disk instead of leaving an empty folder.
• Archived projects can now be deleted. Deleting an archived project will clear any remaining database references to the project and should clear up more space on disk. A deleted archived project cannot be un-archived. For more details about archiving, see the Data Management Guide.
• Projects that are archived, detached, or deleted can no longer be shared with other users.
• Reduced default RAM requirements for some utility jobs.
• Jobs will provide a warning before starting if the amount of RAM available is less than the amount of RAM requested by the job.
• Jobs will no longer fail on completion if they cannot generate their passthrough outputs properly. They will instead raise a warning and continue to completion with potentially missing passthroughs.
• Running Export Exposures from CryoSPARC Live sessions now provides additional information about curation thresholds that were set in the session.
• When maintenance mode is active, a banner will display in the header when viewing CryoSPARC Live sessions communicating to users that additional jobs or sessions will not be queued.
• Ability to specify an arbitrary number of pre-processing GPU workers for a CryoSPARC Live session.
• The Curate Exposures interactive job can now load micrograph thumbnails that were generated in CryoSPARC Live.
• Imports in Import 3D Volumes are now sorted alphabetically.
• Volume Tools now reports the centre of mass of the input volume, if it is a mask.
• Volume Alignment Tools now has the option to automatically re-centre the input volume and particles based on the centre of mass of the supplied mask, especially useful for local refinements.
• Local Refinement’s fulcrum coordinates and Volume Alignment Tools’ 3D centre coordinates now can be specified with units of pixels (px) or Angstroms (A).
• Volume Alignment Tools’ 3D rotation angles can now be specified with units of radians (rad) or degrees (deg).
• Class Probability Filter now supports the new alignments3D_multi particle field.
• Rebalance 2D Classes now always produces a “Particles Selected” output, containing selected particles from all of the superclasses.
• Added parameters Use spherical mask, Spherical mask diameter (A), and Spherical mask diameter outer (A) to allow users to customize the spherical mask used in Heterogenous Refinement. The default behaviour is still the same.
• Orientation Diagnostics will now use the mask_fsc_auto mask present within the input volume connection, if no separate mask input has been connected.
• 3D Flex Generator now outputs a JXX_particle_flows.npy file that saves the trained mesh deformation (flow) of each particle.
• New Freeze latents during training parameter in the 3D Flex Training job allows 3D Flex to better preserve the 3DVA motions when trained from 3DVA latents.
• Inspect Particle Picks now shows the picking statistics histogram in the job’s event log when the job completes, including a border showing the selected thresholds.
• The exposure viewer in interactive jobs and CryoSPARC Live always pans when holding down the middle-mouse button, enabling navigation around the exposure while performing an action such as manual picking or measurement.
• Refinements with global pose search now throw warnings if symmetry expanded particles are connected.
• Reduced default workers and threads parameters for Topaz Preprocessing job.
• Correctly set CPU resource requirements for Topaz jobs based on number of CPUs and number of workers specified in parameters.
• Added a Downsampling mode parameter to Topaz jobs with options of “manual” or “auto”. If set to manual, the user enters Downsampling factor as before. The default is now auto, where the user enters Estimated particle diameter (A) and the downsampling factor will be automatically calculated based on the type of model used.
• Exposure Group Utilities in “cluster&split” mode now respects the “Starting Exposure Group ID” parameter.
• New cluster_vars argument for Job.queue method for queueing to clusters.
• Allow copy or symbolic link operations in project directories for files outside the project.

Fixes

• Previously, refinement jobs recorded internal runtime summary statistics in the database at every iteration. Stats from intermediate iterations are unnecessary and are no longer generated, and existing intermediate stats in the database will be removed in a migration upon updating to v4.5. We expect this change to noticeably improve performance in the web app when viewing or interacting with jobs due to the removed data’s size.
• Repetitive warning messages from libtiff during normal reading of .eer movies have been suppressed, reducing the size of the job log.
• Fixed a bug where ab-initio volumes could not be deselected after being selected in CryoSPARC Live sessions.
• Fixed an issue where it was possible to detach projects that still had jobs queued in the scheduler.
• Fixed a bug that could cause motion correction and CTF estimation jobs to freeze forever if a child process terminates abnormally.
• Prevent applying previously downloaded patches that don’t apply to the current version of CryoSPARC.
• Fixed issue where thumbnails for Import Movies might not zoom in properly
• Fixed a bug introduced in v4.4 that would cause Patch Motion Correction jobs to fail if cleared and re-run, due to the missing input for empirical dose weights.
• Patch Motion Correction now correctly updates the frame count in exposure metadata, in case it was recorded incorrectly by the import job (e.g. if “skip header check” was used). This will help Reference Based Motion Correction to pre-screen its input movies for those with mismatched frame counts, potentially averting a crash.
• Allow using Job Cart with Reference Based Motion Correction.
• Fixed issue which could cause Local Motion Correction to crash at the end of the job while computing summary statistics, on an otherwise successful run.
• Fixed issue where particle diameter was not being calculated correctly in Manually Curate Exposures jobs with connected particle inputs.
• Resolved a bug in Inspect Particle Picks where particle picks from denoised micrographs would report NCC scores outside of [0,1], causing them to be inaccessible to filter.
• Fixed a bug in 2D Classification where if hard classification is enabled, particles would be assigned posterior scores of 0, causing particle counts to be incorrect in Select 2D and subsequent jobs.
• Added override parameter for micrograph pixel size in 2D Classification to fix an issue where duplicate particle removal used the wrong micrograph pixel size on particle datasets from before version 4.5.
• Fixed 2D Classification plots to be consistent with Select 2D Classes job.
• Fixed an issue in Helical Refinement where input scales were previously reset to 0 if greyscale estimation is disabled.
• Fixed plotting bug in 3D Classification which resulted in the focus mask being shifted from its true location by one pixel along each axis.
• Global CTF Refinement now throws an explicit error when the maximum and minimum fit resolutions are incompatible.
• If inconsistent aberration values within a single exposure group are found, refinements with global CTF refinement enabled now fail at the start of the job, rather than midway through.
• Fixed bug that would prevent 3D Variability Display from skipping reconstruction in cluster mode.
• Removed the download PDF option for cluster plots in 3D Variability; the PDFs take a long time to open due to the large number of points being plotted.
• Fixed an issue in which .seg-format 3DFlex mesh segmentation files with certain topologies would fail.
• Fixed an issue in which .mrc-format 3DFlex mesh segmentation files skipped the first segment.
• Particle Sets Tool now correctly uses both a particle’s file path and its index within the file to identify matches when intersecting by path.
• Fixed an issue where Symmetry Expansion and Volume Alignment Tools would ignore any additional particle blob slots (including those produced by Flex Data Prep, such as blob_train).
• In Simulate Data, fix to computation of the additive noise variance. The added noise no longer erroneously depends on the chosen CTF parameters.
• SSD cache system no longer fails when some input particle files are symbolic links that resolve to the same file.
• Prevent invalid multi-instance access when the environment of one instance is active while running cryosparcm commands for another.
• Fixed a bug which caused the Gctf wrapper job to fail if LD_LIBRARY_PATH was unset.
• Normalize project directory names when creating or attaching to avoid duplicate entries in the database.
• Correct Benchmark job directory access when there is a trailing slash in the specified benchmark data directory.
• Restrict accidental cloning of certain Live-managed jobs.
• Fixed an issue where event log PDF downloads for certain jobs (eg. Manually Curate Exposures) would fail due to control characters being interpreted incorrectly.
• Fixed an issue where the interactive scatter plot for 3D Variability Display would not update its layout when the window resized.
• Job PDF report now states the version of CryoSPARC the job was run with, not the version of CryoSPARC running when downloading the report.
• Resolved an issue where job output image previews did not display on the ‘Outputs’ tab when previewing a job.