Electric vehicles (EVs) always integrate into the utility grid via power electronics interfaces like voltage source converters (VSCs). The VSCs manage the power flow between the utility grid and the EVs in bidirectional mode depending upon various conditions. However, the smooth transition of the bidirectional mode depends upon the controller used to drive the references of the VSC. In this paper, the transient performance of the VSC in vehicle-to-grid (V2G) and grid-to-vehicle(G2V) EVs application is tested under voltage sag/swell using the fractional order sliding mode control (FOSMC). The primary focus is to manage the power flow between the utility grid and the EVs based on the state of charge (SOC) of the EV battery and the ampacity of the utility grid under grid transients. A simulation model with a utility grid, VSC, and an EV battery has been modeled at 180kVAR, 400V, and 50 Hz distribution feeder. The stability analysis of FOSMC has been ensured with the Lyapunov candidate function. The results of the proposed control have been compared with the classical PI control. It has been noticed that the proposed control is robust in terms of speedy tracking, fast convergence, and the finest damping.