“Gavin Brown’s enterprise,” as the gallery is dubbed, is in many ways an island, as far as the greater New York art world is concerned. But, in two weeks, that island is set to double in size. With his landlord and co-tenant, famed downtown butcher Pat LaFrieda, decamping to New Jersey, Mr. Brown has finally gotten his hands on the back half of the building. He’ll inaugurate it May 1 with a conceptual show by New York art star Jonathan Horowitz, “Go Vegan!”, that will hang works in LaFrieda’s old butchery and dry-aged coolers.

The gruff and reticent dealer, 46, admitted it’s something of a dream come true: “I have been looking at it for a while—since I moved in, really. I never assumed it would become available but obviously I imagined it.”

The expansion will increase the footprint, and the power, of an art dealer already influential, wealthy and under fire both for too-close ties to the New Museum and for operating something (depending upon which side of it you are on) of an exclusionary art-world clique. But another development in May is also poised to push Mr. Brown more into the spotlight: If that old adage is true—when one door closes, yet another opens wide—then someone stands to gain something in gallerist and Soho impresario Jeffrey Deitch’s pending exodus from New York. And a lot of people are betting the heir apparent is Gavin Brown.

On that Saturday, the casually dressed, heavily bearded dealer was holding court upstairs in the gallery loft where he sometimes presides over cozy, post-opening dinners. As he talked about his gallery and artists, he was demure and more than a little oblique. He became enlivened only when speaking about recent projects (such as Jeremy Deller’s Conversations About Iraq at the New Museum). And, he stressed, he is not such a fan of the sort of hierarchical terms and distinctions divvying up the art world today. “I don’t really know what a gallery is—it’s just a space for potential imagination.” His role sometimes, he explained to The Observer, is less gallerist than “lobbyist.”

Friend and colleague Tom Eccles accused the dealer of having something of an “outward obnoxiousness.” But “he’s actually a bit of a soul brother at heart,” Mr. Eccles, of Bard College, added. “He needs to have artists whom he really feels close with, and he will stick with them long term. It’s different with Jeffrey.… Gavin is a lifer. With Gavin, everyone grew up together—he and his artists.”'

There are vast differences between the two men (and in their respective operations), of course. If Mr. Deitch, the constant, even visionary purveryor of hip, throws the art world’s parties, Mr. Brown runs a private club. Not exclusive, per se, but (and much like his program) tight-knit, with openings that are intimate, almost uncomfortably so. Mr. Deitch shows a slew of artists, some of whom he doesn’t even formally represent; Mr. Brown has had relationships with many of his for years. One man is famous in the broader world; Mr. Brown has long flown, by choice, far lower on the radar.

But there are similarities, too. Over the past 15 years, the British “dealer” has built a reputation that is predicated on close working relationships with the artists he represents, some of whom have stuck with him since the very beginning, Peter Doig, and Elizabeth Peyton among them. (Notably, superstar Chris Ofili, whom Mr. Brown represented during the high-profile “Sensation” controversy nearly a decade ago, has moved to David Zwirner.) Through the years, Mr. Brown enabled their whims and fancies—be it in his own gallery (Urs Fischer’s ground-excavating You, 2007) or in someone else’s (Rob Pruitt’s freewheeling “Art Awards” at the Guggenheim last year). He helped many of them earn institutional attention and support; the Museum of Modern Art, for instance, has acquired no less than 16 Doig works and about 30 Peyton drawings and paintings.

And like Mr. Deitch, who declined to comment for this story, Mr. Brown knows how to create a scene. His Passerby art bar in the meatpacking district was the establishment for much of its eight-year existence. And for now, he has no plans to reopen. But even though Mr. Brown’s bar-back days might be over, his program remains intrinsically social. Mr. Pruitt’s “Art Awards” was but one of these endeavors—in the sendup-of-a-gala gala, various members of the Brown clique gave awards to each other and friends. The Guggenheim confirmed that it will host a second installment in December.

“It feels like he’s everywhere,” said Alex Zachary, a former GBE director who recently jumped ship to open his own gallery on the Upper East Side. 

Mr. Brown admitted that Mr. Deitch’s departure for Hollywood, to run the Los Angeles Museum of Contemporary Art, got him thinking. “For any person who owns a gallery in New York, the news of that was sort of ‘Huh … could I do that?’” he told The Observer. “I thought about it and came to the conclusion that no, I probably shouldn’t. Probably couldn’t do it—I’m not sure I’d feel my life would be my own anymore. I’d probably make a complete hash of it.” He’s also pretty tied to New York, he noted: The father of three is in the midst of moving to a new home in Harlem.

“I think one of the best aspects of a gallery is its very intimate relationship with the artist,” Mr. Brown said. “The process is much faster and more immediate. You have a one-to-one, almost simultaneous translation [that can’t happen in a museum]. I suppose it is actually a thrill and kind of a privilege to be there with culture as it happens.” 

This, of course, is not exactly a charitable venture. Mr. Brown represented Mr. Doig’s work, for example, when it was selling for a few thousand dollars; in 2007, the artist’s White Canoe set an auction record for a living artist when it brought $11.3 million. But, ironically, the man whose Internet address is gavinbrown.biz doesn’t like talking about money.

Mr. Brown’s program has been so successful, in fact, that harsh criticism arose last year over the dealer’s relationship with the New Museum (i.e., “Gavin’s Place,” to the naysayers), where four of his artists have had substantial showings in the past two years. Brown outright denied any insider politics (as did the New Museum at that time, though the institution declined to comment for this piece).

“I’m not sure I ever saw what was so controversial,” he bristled of the much-rehashed issue. “I just don’t buy into that cynical way of seeing it. In terms of my role in the supposed controversy, you could see me as a lobbyist. That’s one way of looking at it. Another way of seeing this is that people, curators amongst them, really believe in the voices of these artists. I don’t see the problem—are people upset about the works in the exhibitions? Do they not want to see that work? The ones who do see a problem are small-minded people keeping score in their small-minded game.”

Mr. Brown ended the interview waxing eloquent. “I wonder about the future and how people will communicate,” he said. “We’re at an epochal moment and I’m just trying to figure out what goes in the trash and what is to be taken up and passed on to future generations.”

When asked to explain, Mr. Brown said he’d rather not.

editorial@observer.com