Handling large displacement optical flow is a remarkably arduous task. For instance, standard coarse-to-fine techniques often struggle to adequately deal with moving objects whose motion exceeds their size. Here we propose a learning approach to the estimation of large displacement between two non-consecutive images in a sequence on the basis of a learning set of optical flows estimated a priori between different consecutive images in the same sequence. Our method refines an initial estimate of the flow field by replacing each displacement vector by a linear combination of displacement vectors at the center of similar patches taken from a code-book built from the learning set. The key idea is to use the accurate flows estimated a priori between consecutive images to help improve the potentially less accurate flows estimated online between images further apart. Experimental results suggest the ability of a purely data-driven learning approach to handle fine scale structures with large displacements.