The Grill Room at The Four Seasons Restaurant is one of the classic rooms in the city, the minimalist Philip Johnson decor striking an exacting backdrop for the power lunches and celeb-heavy dinners that dot the landscape of tables. And until recently, it was the home of a giant Picasso the 19-by-20-foot tapestry Le Tricorne. After 55 years at the restaurant, Aby Rosen—the real estate tycoon and major collector who owns the Seagram Building, where the Four Seasons is housed—decided it had to go. Some shouting matches ensued, but mr. Rosen finally got his way, and the painting was gone.