Actor James Cann and Playboy founder Hugh Hefner have a laugh during Fight Night at the Playboy Mansion on July 15, 2003 in Bel Air, California.

James Caan and Hugh Hefner in 2003

| Credit: Donald Miralle/Getty

James Caan once had an extended stay at the Playboy Mansion.

Back in 2003, the actor — who died Wednesday at the age of 82 — told Esquire, "To get over my divorce, I got a prescription to live at the Playboy Mansion for a while."

James was previously married four times: first to Dee Jay Mattis, from 1961 to 1966, then to Sheila Ryan in 1976. He later tied the knot with Ingrid Hajek in 1990. The two divorced in 1995, and he wed Linda Stokes later that year.

The actor filed for divorce from Stokes in 2005, but the two later reconciled. James and Stokes were in the middle of a divorce as of December 2016, according to TMZ.

"I have four wives and five kids. I apparently don't know the difference [between sex and love]," James joked in his 2003 Esquire interview.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.

Actor James Caan, Scott Caan and family attend the premiere of "Misery" on November 29, 1990 at Mann Village Theater in Westwood, California.