We present TransProteus, a dataset, and methods for predicting the 3D structure, masks, and properties of materials, liquids, and objects inside transparent vessels from a single image without prior knowledge of the image source and camera parameters. Manipulating materials in transparent containers is essential in many fields and depends heavily on vision. This work supplies a new procedurally generated dataset consisting of 50k images of liquids and solid objects inside transparent containers. The image annotations include 3D models, material properties (color/transparency/roughness...), and segmentation masks for the vessel and its content. The synthetic (CGI) part of the dataset was procedurally generated using 13k different objects, 500 different environments (HDRI), and 1450 material textures (PBR) combined with simulated liquids and procedurally generated vessels. In addition, we supply 104 real-world images of objects inside transparent vessels with depth maps of both the vessel and its content. We propose a camera agnostic method that predicts 3D models from an image as an XYZ map. This allows the trained net to predict the 3D model as a map with XYZ coordinates per pixel without prior knowledge of the image source. To calculate the training loss, we use the distance between pairs of points inside the 3D model instead of the absolute XYZ coordinates. This makes the loss function translation invariant. We use this to predict 3D models of vessels and their content from a single image. Finally, we demonstrate a net that uses a single image to predict the material properties of the vessel content and surface.
翻译:我们展示了TransProteus, 一个数据集, 以及用来预测三维结构、 材料、 液体和物体的3D结构、 遮罩和特性的方法。 透明容器中的合成( CGI)部分是程序性生成的, 使用13k 不同的天体、 500个不同的环境( HDRI) 和 1450 个材料纹理( PBR), 与模拟液体和程序性生成的船体相结合 。 此外, 我们提供由程序生成的由50k 个液体和固态物体图像组成的新的数据集, 在透明容器内 透明容器内 。 图像说明包括 3D 模型、 材料属性( 颜色/ 透明度/ 深度... ) 和 3D 容器及其内容的分隔面罩。 数据集的合成( CGI) 部分是按程序生成的, 使用13k 不同的天体、 500 不同的环境( HD) 和 1450 材料质质体中材料( PBR) 。 此外, 我们提供104 真实的物体图像世界图像图像图像图像图,, 我们使用X 的精确的图像 的解算。