This is a step by step center/gearshift surround trim removal guide for the swifts from 2018 onwards.
First step is to jack up your car. Just kidding, you don't need to. Move the front seats all the way to the front

and sit in the back seat.
Remove this plastic rivet by any means possible. This is not a push type rivet and honestly I have no idea how to remove it. If somebody does, please let me know. There's one more on the other side.
I removed it elegantly by using a circlip removal tool.
Get out and slide the driving seat back and sit on it. Now you need to 'pop out' and 'slide' this panel (exactly in this order). There's one more on the other side.
Next step is to pull up on this panel. You don't need anything to pry this out if you start from the front, like I did. It comes out easily.
After removing the above panel, you'll need to remove the oddly shaped cup holders. Pry it out using a screw driver.
After lifting out the cupholder you'll gain access to 2 philips head screws. Kindly remove them.
Now sit back. Remove the shiny Philips head screw that was supposed to be in that red circle. Sorry, I forgot to take a picture before removing it.
Next step is to remove the small plastic panel under the handbrake lever. There's a small gap between the handbrake lever and the panel, push a finger through it and gently pull it out.
Now you're treated with another plastic rivet. The same kind is used to fit the door pad to the door.
You need to slide the center console under from this clip. To do that, hold the console with both hands on marked locations and pull towards you (make sure you're sitting in the driver seat).
And that's it. Now the center console is free. Lift the whole thing up, starting from the back.
Hope this help someone. Thanks for reading.