Machine learning creates a 3D model from 2D images – Source

Researchers from the McKelvey School of Engineering at Washington University in St. Louis have developed a machine-learning algorithm that can create a continuous 3D model of cells from a subset of 2D images taken using the same standard microscopy tools found in many labs today. .

The intelligence of nature’s machine.

“We train the model on a set of digital images to get a continuous representation,” he said. Ulugbek KamilovAssistant Professor of Electrical and Systems Engineering, and Computer Science and Engineering. “Now, I can show it any way I want. I can zoom in smoothly and there are no pixels.”

Key to this work was the use of a neural field network, a specific type of machine learning system that learns to map from spatial coordinates to corresponding physical quantities. When the training is completed, the researchers can indicate any format and the model can provide the value of the image in that location.

One of the particular strengths of neural field networks is that they do not need to be trained on copious amounts of similar data. Alternatively, as long as there are enough 2D images of the sample, the network can represent it completely, both inside and out.

The image used to train the network is like any other micrograph. In essence, a cell is lit from below; Light travels through it and is captured on the other side, creating an image.

The imaging system can enlarge a pixelated image and fill in the missing parts, creating a continuous 3D representation.

“Because I have some views of the cell, I can use those images to train the model,” Kamilov said. This is done by feeding the model information about a point in the sample where the image captured some of the cell’s internal structure.

Then the network takes the best of it to recreate that structure. If the output is wrong, the network will be modified. If true, this path will be strengthened. Once the predictions match real-world measurements, the grid is ready to populate parts of the cell that weren’t captured by the original 2D images.

The model now contains information on a complete and continuous representation of the cell – there is no need to save a data-heavy image file because it can always be recreated by the neural field network.

Not only is the model a true representation of the cell that’s easy to store, Kamelov said, but it’s also, in many ways, more useful than the real thing.

“I can put any formatting and generate that view,” he said. “Or I can create entirely new shows from different angles.” He can use the model to rotate a cell like the top or zoom in for a closer look; use the form to do other numerical tasks; Or even enter it into another algorithm.


This work was supported by the National Science Foundation, Awards CCF-1813910, CCF-2043134, CCF-1813848, and EPMD-1846784

The McKelvey School of Engineering at Washington University in St. Louis encourages independent research and education with an emphasis on scientific excellence, innovation, and collaboration without borders. McKelvey Engineering has high-level research and graduate programs across departments, particularly in biomedical engineering, environmental engineering, and computing, and has one of the most selective undergraduate programs in the country. With 140 full-time faculty members, 1,387 undergraduates, 1,448 graduate students, and 21,000 live alumni, we are working to solve some of society’s biggest challenges; To prepare students to become leaders and innovate throughout their careers; and to be a catalyst for the economic development of the St. Louis region and beyond.

Leave a Comment