This paper presents a residual DCT matrix based MPEG-2 to H.264/AVC inter transcoding algorithm to meet the requirements of real-time transcoding in some special environments such as streaming media. In the proposed inter transcoding algorithm, the re-encoding mode is decided by the edge character of the residual DCT matrix, and the selection of search center and search range in the H.264 motion estimation process is based on the re-encoding mode. Experimental results show that the proposed method reduces about 2/3 calculation complexity with the cost of 5.7% bit-rate increase and 0.08dB PSNR decrease. The most important point is that there are not any preset or posterior parameters, which is especially useful for real-time transcoding.