Registration Of Deformed Tissue AGNN-VAE Approach With Data Assimilation For Sim-to-Real Transfer

In image-guided surgery, deformation of soft tissues can cause substantial errors in targeting internal targets, since deformation can affect the translation of preoperative image-based surgical plans during surgery. Having a realistic tissue deformation simulator could enhance the accuracy of internal targets localization by giving an accurate estimation of the deformation applied to a preoperative model of the organ. A key challenge is to address the sim-to-real gap between the simulator and the actual intraoperative behaviour of the tissue. The sim-to-real transfer challenge is addressed by formulating the problem as a probabilistic inference over a low-dimensional representation of deformed objects. The proposed method utilizes a generative variational autoencoder structure based on graph neural networks (GNN-VAE) to generate a probabilistic low-dimensional representation of the outputs of a physics-based simulator. To match simulation data to real data, the resultant low-dimensional distribution (i.e., prior distribution) is updated iteratively using an Ensemble Smoother with Multiple Data Assimilation (ES-MDA). The advantages of the proposed method are 1) it only uses simulation data for training the GNN-VAE, and no retraining of GNN-VAE is required intraoperatively, 2) it does not require estimating the mechanical properties of the tissue it is simulating, and 3) is able to work with any physic-based simulator. The proposed framework was verified both in experimental and simulation studies and showed it can reduce the registration error in tissue deformation.