By Jamie Gangel, Sunlen Serfaty and Eric Bradner, CNN

(CNN) -- Sources in the Senate tell CNN they have been worried about Sen. John McCain recently, following a week when the Arizona Republican was kept from the Senate to address side effects from his brain cancer treatment.

The sources described McCain as looking increasingly frail and said he has not spoken up in recent GOP meetings the way he had before, in addition to his absence this week for treatment at Walter Reed Medical Center.

One source said that McCain always used to speak up in these meetings, and that he hasn't at all recently. The source was not commenting on McCain's mental acuity, but on his energy level, and pointing out that his lack of participation was not normal. One well known side effect of cancer treatments is fatigue.

Sen. Lindsey Graham of South Carolina, McCain's closest friend in the Senate, said he spoke with Cindy McCain on Wednesday about her husband's health.

"He is receiving treatment for the side effects of therapy," Graham said. "I feel pretty good about the way the treatment is affecting his underlying cancer. But the treatment has a downside. So he is trying to rest up. I am very confident that he will come back and continue to participate for a long time to come."

Graham said he did not talk to McCain's wife Cindy McCain about the potential for the senator to return next week for a tax vote; he only got an update on his health. He said he hopes to talk to McCain Thursday or see him in person soon.

"The main thing I am focused on is to just get better. John, take a little time. Rest up. It's OK," Graham said. "Take a day or two off. He's had a heck of a Senate schedule. And I hope he will take some time to regroup, do a little rehabilitation on his leg, and come back and participate in a way that only Sen. McCain can participate."

McCain received medical treatment last month for a minor tear to his right Achilles tendon requiring him to wear a walking boot.

House Speaker Paul Ryan said Thursday that GOP are being "flexible" on the timing of final votes on a major tax reform bill due to health-related absences from the Senate, and said he has spoken with Senate Majority Leader Mitch McConnell about timing.

"We're simply being flexible to honor their concerns about managing their schedule and some possible absences," Ryan said.

McCain, 81, is five months into treatment for brain cancer. He was diagnosed in July with a primary glioblastoma, a type of brain tumor. He underwent surgery to remove a blood clot associated with the tumor at the Mayo Clinic in Phoenix, and lab results from that surgery confirmed the presence of brain cancer associated with the blood clot.

McCain returned to Walter Reed this week for treatment that his office said was related to side effects of his cancer therapy.

Arizona Sen. Jeff Flake said he too spoke to Cindy McCain on Wednesday night about her husband's health.

"We expect him back," Flake told reporters. "I expect him back. He's obviously recovering from the effects of the treatment and so we expect him back."

Throughout his treatment, McCain has kept up his reputation as the "maverick" of the Senate -- which was on display in July, when McCain memorably returned from cancer treatment to walk onto the Senate floor and cast his thumbs-down vote against the Republican effort to repeal parts of President Barack Obama's signature health care law.

McCain has been a thorn in the side of President Donald Trump -- who'd criticized McCain in 2015, questioning McCain's war hero status and saying he prefers those "that weren't captured."

In a speech in Philadelphia in October this year, McCain fired back at Trump with a complete rejection of Trump's view of the United States' role in the world.

"To fear the world we have organized and led for three-quarters of a century, to abandon the ideals we have advanced around the globe, to refuse the obligations of international leadership and our duty to remain 'the last best hope of earth' for the sake of some half-baked, spurious nationalism cooked up by people who would rather find scapegoats than solve problems is as unpatriotic as an attachment to any other tired dogma of the past that Americans consigned to the ash heap of history," McCain said then.

As recently as Wednesday, McCain delivered a rebuke of Trump via Twitter, urging him to lay off his claims of "fake news" and saying Trump "must understand his harmful rhetoric only empowers repressive regimes to jail reporters & silence the truth."

Glioblastoma is an aggressive tumor that took the life of Beau Biden, the son of former Vice President Joe Biden, in 2015.

Biden spoke emotionally with McCain's daughter, Meghan McCain, on ABC's "The View" on Wednesday, recalling how her father had helped watch over his sons Hunter and Beau decades earlier.

"Look, one of the things that gave Beau courage, my word, was John (McCain). Your dad, you may remember when you were a little kid. Your dad took care of my Beau," Biden told Meghan McCain, holding her hand.

He talked about how the two had run against each other in the 2008 presidential race.

"We're like two brothers who were somehow raised by different fathers or something because of our points of view," Biden said. "Even when your dad got mad at me, said I should get the hell off the ticket, and remember what I said about your dad? I said, I know, and I mean this sincerely, I know if I picked up the phone tonight and called John McCain and said, ... 'I need your help, come,' he'd get on a plane and come. And I would for him."

The-CNN-Wire

™ & © 2017 Cable News Network, Inc., a Time Warner Company. All rights reserved.