Yes, a king could abdicate. We don’t have a clear case of one actually abdicating in Westeros as far as I can recall (though plenty of kings abdicated in real life), but we do have the abdication of Prince Duncan the Small who chose to renounce his claim to the throne rather than give up Jenny of Oldstones. Maester Yandel also mentions forced abdication as a possible method for a Great Council to remove Aerys in his account of the stories told about the true purpose of the tourney at Harrenhal. Robert Baratheon, while not a Targaryen king, also mentions his dream of giving up the throne and leaving for the Free Cities to Ned and how the only thing stopping him was the thought of Joffrey on the throne.
But just because a king could abdicate does not mean that Aerys would. Why would he? Bookishness does not necessarily translate to a desire to be a maester, and we hear nothing of a noted piousness on his part that could suggest he was interested in becoming a septon. Note that joining either order means accepting their rules and limitations, forswearing his name and titles for a life of service, and committing to their studies, all of which he probably had no interest in doing. Aerys wasn’t just bookish, he was specifically concerned with magical arcane texts and lore, the exact kind of studies that are not really supported or encouraged by the Citadel. Neither would his study there even offer him access to the rare texts that only a handful of senior archmaesters would have access to. So, what’s the upside of abdicating and becoming a maester for him? Aerys was already getting what he wanted; he was the highest authority in the land and no one could force him to do something against his will. He refused to consummate his marriage, refused to produce an heir, refused to be a proper ruling monarch and left running the country to Bloodraven while he kept company with arcane texts and lost lore that he perused at his convenience and leisure. No one could restrict his reading or prevent him from pursuing the texts he wanted, and he had a fellow magically-minded individual in Bloodraven to support his beliefs. Why give it up and submit himself to the authority of an order resistant to his subject of interest to control what he does or studies, or enter into useless arguments with conservative maesters who don’t believe in any of that prophecy and Others crap?
Too, I’d raise the question about the consequences an abdication would have brought to the realm. One of the major criticisms leveled at Aerys is that his refusal to consummate his marriage could have jeopardized the realm because it left Prince Rhaegel as the heir to the throne. Rhaegel was repeatedly described as a sweet man but also as “mad, meek and sickly” and was known for dancing naked in the Red Keep. In refusing to have a child, Aerys showed that he was willing to let a woefully unfit prince succeed to the throne at a time when the Seven Kingdoms were facing threats from within and without, not that Aerys cared much about that as his abandonment of the Starks and Lannisters to face Dagon Greyjoy on their own showed. Still, an abdication would have put Rhaegel on the throne which would have come with its own set of problems.