Mr. Cowles has covered one long wall of the loft with photographs, arranged “semi-biographically,” he said rather than in any kind of linear narrative. They are a collection of memories and art that he loves: a portrait of his stepfather, media mogul Gardner “Mike” Cowles Jr.; one of his old friend, the artist David Hockney; an autographed Ed Ruscha photo.  Bearded, balding and beaming, Mr. Cowles is himself an amateur photographer who goes by the pseudonym Charlie Snaps.

Now 70, he bought his first work of art at 18: a Cezanne print from a shop on Madison Avenue. His next major purchase, at 20, was a Tom Holland painting. Since then, he has collected hundreds of pieces of fine art for himself and his gallery.

Mr. Cowles grew up at the fabled building of 740 Park Avenue. His mother, Jan Hochstraser, married Mike Cowles Jr. , whom Mr. Cowles refers to as “Dad,” in 1956. At his most successful, the senior Mr. Cowles owned 50 newspapers and somewhere between 40 and 50 magazines, including Look, which launched a year after Life. Mr. Cowles rattled off the celebrities he met as a child: Marilyn Monroe, J.F.K., Audrey Hepburn. “My parents liked to entertain, and everybody who was anybody in the world was invited,” he said.  Celebrity gossip columnist Hedda Hopper was particularly friendly to young Charles, who owns two photographs of a manic Ms. Hopper shrieking after a pigeon landed in her hat.

Mr. Cowles worked in the photo archives of Look the summer he was 14. Once, he was shown a picture of  Monroe sitting on a park bench reading the paper with a canoodling couple to her left. Mr. Cowles’ editor dispatched him to find their identity. First he had to find Marilyn. “I said, ‘how the hell do I do this? I called a celebrity service to find out where she was staying; I went around to see all the doormen near her hotel.” Mr. Cowles eventually found out that the woman from the necking couple was married, and her companion in the picture was not her husband. Look refused to print it. Two years ago at Sotheby’s, the photograph came up for auction. It now hangs on the wall of photographs in Mr. Cowles’s loft.

The gallerist said he felt no pressure to take the reins of the Cowles family publishing empire. He majored in journalism at Stanford, but had always been influenced by his step-uncle, Russell Cowles, a modernist painter. Charles interned at the San Francisco–based art magazine Artforum during his time as an undergraduate, working on advertising and circulation. Artforum, he said, was “the bridge between my interest in the art world and my interest in publishing.”

At the end of the internship, Artforum’s publisher told Mr. Cowles that the magazine was broke and would have to fold. Mr. Cowles told the publisher, ‘I can make it profitable, but you have to go.’ He did. Mr. Cowles sold the magazine for one dollar, “helped to stabilize” the publication, and worked there for just shy of a decade.

He was then  asked to be the first curator of modern art at the Seattle Art Museum. It was no Soho. The Museum’s director was primarily interested in Asian art that “he could hold in his two hands, like tea bowls,” Mr. Cowles said, and the cosmopolitan curator”s first proposed acquisition of Andy Warhol’s Double Elvis was dismissed as “too big.” 

Mr. Cowles moved back to New York in 1979, purchased gallery space at 420 West Broadway, formerly owned by Leo Castelli, and opened his eponymous gallery in May 1980. The stated mission was to “stray away from the well-established artists who already had New York galleries,” he said. “If I got artists who didn’t live in New York, they wouldn’t be hanging about the gallery too much bothering us.”

Mr. Cowles admitted that he originally would have loved “to show Lichtenstein or Warhol”—he was friendly with both men, particularly the former.  Prints of women by the two artists cast sidelong looks at each other opposite his vast living room.

In 2000, the gallery moved to West 24th Street, part of what Mr. Cowles called “the overnight migration” of galleries among his friends and colleagues from Soho to Chelsea.

As soon as he moved in, Mr. Cowles began to look ahead. “O.K., I’ll be 70 ten years from now, that’s when I want to retire,” he said. “So then 70 rolled around and I said, ‘All right, we’re closing.’” Then the recession hit. “We’ve lost a lot of money the last two years at the gallery, and I just said, ‘I’m tired of this.’”

Mr. Cowles repeated often that he had no regrets. “I’ve had a really fabulous life,” he said quietly. And how does he feel now?

“How about liberated?” Mr. Cowles said defiantly. “With the amount of art world life there is in New York, I’m not going to be out there in the cold.”

 Mr. Cowles is balding and sports a white, Hemingway-esque beard. He speaks in a growl but smiles often.