Steel has played a leading role in the development of human civilization and technology. To achieve the goal of industry 4.0 for steel industry, intelligent automated manufacturing becomes increasingly important. One of the key steps on the automatic production line of steel pj-oducts is automatic inspection of surface detects. In this paper, we propose a fast vision-based surface inspection framework for defects on continuous casting steel billets. The proposed method relies only on simple pixel-domain vision-based operations to achieve fast and accurate inspection of surface defects. Experimental results have demonstrated that our method efficiently achieves good inspection accuracy on corner and sponge cracks on steel billets.