High-sensitivity detection of microscopic magnetic field is essential in many fields. Good sensitivity and high spatial resolution are mutually contradictory in measurement, which is quantified by the energy resolution limit (ERL). Here we report that a sensitivity of 0.5 nT/\sqrt{Hz} at the nanoscale is achieved experimentally by using nitrogen-vacancy defects in diamond with depths of tens of nanometers. The achieved sensitivity is substantially enhanced by integrating with multiple quantum techniques, including real-time-feedback initialization, dynamical decoupling with shaped pulses, repetitive readout via quantum logic. Our magnetic sensors will shed new light on searching new physics beyond the standard model, investigating microscopic magnetic phenomena in condensed matters, and detection of life activities at the sub-cellular scale.