Reconstructing the 3D geometric model of buildings from photogrammetric images, from LiDAR point cloud, or from both of them has raised extensive researches in the past decade. However, the façade mapping is still the bottle neck in the photo-realistic building modeling process. With the rapid development of personal mobile computing devices, such as smartphones and tablet PCs, it has become easier to directly geo-referencing the taken picture. The concept of a semi-automated approach to modeling photo-realistic 3D buildings using a personal mobile computing device is proposed in this paper. The built-in GPS receiver provides approximate 3D coordinates while the G-sensors deliver three rotation angles of the camera at the shooting moment. However, these parameters are not precise enough for the further photogrammetric operations. Therefore, a model-image fitting algorithm is necessary to determine precise image orientation. By introducing the "Floating Models" concept, the object modeling and image orientation problem can be solved efficiently through the semi-automated procedures based on the Least-squares Model-image fitting (LSMIF). Since the object model can be extracted and the photo orientation can be determined, the creation of realistic texture image, which is also called inverse mapping, can be automated by coordinate transforming and image resampling. Finally, a photo-realistic building 3D model is reconstructed through these semi-automated procedures.