Steel production requires high-quality surfaces with minimal defects. Therefore, the detection algorithms for the surface defects of steel strip should have good generalization performance. To meet the growing demand for high-quality products, the use of intelligent visual inspection systems is becoming essential in production lines. In this paper, we proposed a ShuffleDefectNet defect detection system based on deep learning. The proposed defect detection system exceeds state-of-the-art performance for defect detection on the Northeastern University (NEU) dataset obtaining a mean average accuracy of 99.75%. We train the best performing detection with different amounts of training data and observe the performance of detection. We notice that accuracy and speed improve significantly when use the overall architecture of ShuffleDefectNet.