Before getting close proximity of the small body, its ephemeris and physical parameters need to be estimated through solely optical measurements (OM) of the spacecraft (s/c), which is known as the early characterization phase (ECP). During ECP, the s/c is navigated by ground-based operations and flies hyperbolic arcs for rapid excursions and less sensitivity to both gravity and orbital insertion errors. Multiple maneuvers are required to change the s/c from one arc to the next. To improve the ECP efficiency, using the optical data from previous missions, e.g. Rosetta, the neural network based autoencoder is applied to learn the correlations between the input images and the Lambertian sphere to produce the best fit and centroid’s position, with the improved execution time and robustness to lens distortions and sensors noises, as well as the increased adaptability to the specific unknown environment. Solely with this high precision OM and given the highly nonlinear and perturbative dynamics, the Unscented Kalman Filter is applied to acquire more accurate estimation of the body’s physical parameters and relative state of the s/c w.r.t the body. Moreover, the number and geometry of the arcs are optimized to minimum the ECP duration and fuel cost and maximize the science return. The navigation performance is improved by applying autoencoder for the first time and multi-objective optimization to design the ECP, though AI algorithms have been applied to the autonomous imaging and mapping of the small body for global mapping and close observation phases. This methodology is tested and validated with the hardware in the loop, to finetune the autoencoder and give practical references to design efficient and robust ECP of small body missions. It can be extended to other missions with similar ECP scenario, e.g. the detection and approach phase of active space debris removal.