Building models are conventionally reconstructed by measuring their vertices point-by-point in a digital photogrammetric workstation (DPW), which is time and labor consuming process. Although aerial photos implicitly provide 3D information of buildings, LiDAR systems directly provide high density and accurate point cloud coordinates. However, LiDAR data cannot accurately represent the building boundaries. To take advantage of both systems, we propose Floating Model and a tailored least-squares model-data fitting (LSMDF) algorithm in this paper. The floating model is a pre-defined primitive model, which is described by a set of parameters, floating in the space. A building is reconstructed by adjusting these model parameters so the wire-frame model adequately fits the building's boundary in all overlapping photos and LiDAR data. The semi-automated modeling procedure consists of 3 steps. First, the operator chooses an appropriate model and approximately fit it to the building's outlines on the aerial photos. Then, an automated procedure computes the optimal fit between the models and both of aerial photos and LiDAR data using an iterative LSMDF algorithm. Finally, the model parameters and standard deviations are provided, and the wire-frame model is superimposed on all overlapping aerial photos for the operator to check or modify the results. To test the proposed algorithm and approach, an image block of 4 panchromatic aerial photos and corresponding LiDAR data are selected for the experiments. The ground resolution of the image is approximately 5cm. The point density of LiDAR point cloud is about 4-5point/m 2. The reconstructed models are manually evaluated and compared. Most of the buildings are accurately modeled, and the fitting result achieves the photogrammetric accuracy. In addition, the implicit constraints within the model, such as the parallel edges or rectangle corners, will keep the building shape without distortion.