Haunted by Giving Up His Baby, Philips Sues Charles Allen Family

James Philips removed his sunglasses and rubbed his red eyes. “This is the bad part,” he said. An embarrassed smile flickered across his lined face and he looked down at the floor. He shook his head. Up until this point, the story that he had been recounting had no shortage of bad parts. Now he wanted to make a distinction.

“I’ve always got this thing in the back of my mind about what happened, you know?” Mr. Philips told The Observer . He had been telling the story of his disastrous marriage in 1952 to Terry Allen. He was 24 years old then and a trader on Wall Street. (His grandmother had purchased his seat on the New York Stock Exchange.) Ms. Allen, daughter of Wall Street enigma Charles Allen Jr., heiress to the Allen & Company fortune, would eventually become a successful theatrical producer ( Sugarbabies , Me and My Girl ) and a social force of nature, but at the time she first married, she was only 18.

Mr. Philips repositioned the sunglasses, wraparounds with chocolate-brown lenses, on the bridge of his aquiline nose. They were ill-suited to a 70-year-old man, but the story he was about to tell was equally out of place in the narrative of a man of his pedigree. “I’ve got a bad, guilty feeling,” Mr. Philips said. “It has haunted me for my whole life.”

What burdens Mr. Philips is a deal that he said he made with his former father-in-law. The agreement, he said, was this:

In exchange for his giving up his parental rights to his 2-year-old daughter with Terry Allen and ceasing all contact with the child, he would be compensated by Charles Allen for a sum that would amount to $2.7 million. When the money eventually stopped coming, Mr. Philips responded by filing suit in State Supreme Court in Manhattan. The suit, which is still pending, alleges that Charles Allen reneged on his deal.

On paper, the union looked like the merger of two wealthy New York families. Mr. Philips’ father, Herman Philips, had built a fortune out of real estate, including majority ownership of the Sherry-Netherland Hotel, where the Philips-Allen nuptials took place. Allen, the intensely private and extremely influential investment banker, was a high-school dropout who had formed Allen & Company in 1922 and, with his brother Herbert Allen, built a family fortune worth hundreds of millions of dollars. Forbes magazine would list him among the “canniest investors of all time.” Allen’s 1994 obituary in The New York Times called him the “shy Midas of Wall Street.”

The marriage between his daughter and James Philips lasted all of six years. Part of that time, Mr. Philips said he carried on an affair with another woman whom he had met while walking the dog in Central Park, outside their Fifth Avenue apartment. The extramarital relationship continued, even after Ms. Allen gave birth to a daughter, Toni, in 1956. In 1958, the couple divorced and Ms. Allen wed Irwin Kramer, who worked at his family’s hotel, but eventually became a partner in her father’s firm.

Mr. Philips seemed ashamed of what he had done as he recounted incidents that had occurred 20, 30, even 40 years ago. Giving up Toni “remains the one thing I truly regret in my life,” he said. But there was also astonishment in his voice, as if he himself could not quite believe the twists and turns of the story he was determined to tell over the course of several conversations with The Observer . “Have you ever heard a story like my story?” he asked at one point. He answered the question himself. “No. No one else has either.”

Terry Allen Kramer was on the phone. Her voice was husky and resolute. “He was a rotten husband and that was it,” she said. “So what. There’s nothing unusual about that.” Mrs. Kramer had refused to meet with The Observer , but she did agree to listen to Mr. Philips’ allegations. (She explained that she might refuse to answer questions, but added, “I would never lie to you.”) Mostly though, she dismissed her ex-husband’s claims and said she had no knowledge of any deal between her father and Mr. Philips. She said that her ex-husband was trying to intimidate her into settling by going to the press. “I’m not afraid of anything,” she said. “I’ve been around long enough to know that that’s what he’s trying to do.” She also said, “Only animals give up their young.”

New Mergers and Acquisitions

Mrs. Kramer confirmed that she and Mr. Kramer broached the idea of adopting Toni, but Mr. Philips said that shortly after that–October 1958, according to the court papers–Charles Allen got involved, initially through agents and then himself. At first, Mr. Philips said, he rejected their entreaties. Still, he said, “they hounded me and hounded me.” He had visitation rights to see his daughter, but he claimed that his ex-wife “made it difficult for me to be comfortable there. She didn’t want me to touch the baby, to take it to the park,” he said, strangely unaware of how he was depersonalizing his child in his choice of pronoun. “When I picked the baby up, she used to try to pull it away from me. The baby would get hysterical.”

In the deposition that he gave in 1993, Mr. Philips said that Charles Allen called him to ask him to consider the adoption for his daughter’s sake. He said then that it was one of only two times that Allen “ever spoke to me, anyway, in a personal way.” The other occurred one night when his former father-in-law called Mr. Philips to tell him that his daughter had a very high fever.

Not long after Terry Allen married Irwin Kramer, Mr. Philips remarried, too, to Patricia Clarke, the woman with whom he had been having an affair during his marriage. “Terry felt that Patsy was the instigator behind the whole split between us,” said Mr. Philips. “She resented it terribly.”

Eventually, he said, “I thought to myself, It probably would be better for the child in the long run,” he said. “Security-wise, certainly. Secondly,” he volunteered that his second wife “was kind of a neurotic person” who at one point was “hooked on all sorts of stuff like sleeping pills.”

Mrs. Kramer said that Clarke, who died young, “never entered my mind.” Mrs. Kramer also claimed that Mr. Philips “couldn’t have cared less about his daughter.” She said that they could not have fought over their daughter because “he never came to see her” after the couple separated.

Then Mrs. Kramer seemed to recall something from that time. “Now it all comes back,” she said. “He said, ‘Give me back the diamond engagement ring I gave you, and I’ll agree to [the adoption].'” Mrs. Kramer said she returned the ring. “Material things don’t mean anything in my life,” she said. “My concern is my children’s welfare.”

In the May 1998 issue of W magazine, Mrs. Kramer cooperated with a 10-page feature on her and “La Follia,” the 46,000-square-foot oceanfront “palazzo” that she and her husband built in Palm Beach (they have also owned homes in Manhattan, Southampton, L.I., and Lyford Cay, Fla., where, according to W , the home is dubbed “the Kramertorium”). Without mentioning Mr. Philips’ name, she said in the article that she’d “made a disastrous first marriage.” She also said that her father “thought there should be no emotions connected with money.”

A Deal Sealed With a Handshake

Mr. Philips’ decision to allow the Kramers to adopt his daughter became more complicated when, he said, his parents discovered the situation. “This was their only grandchild,” he said. “My father was horrified that we had gotten divorced, and my mother was close to Terry. They loved the child.” Mr. Philips said that his father threatened to disown him if the adoption occurred.

Mr. Philips told The Transom that he then sat down with Allen and his attorneys to explain his predicament. According to court papers filed by Mr. Philips’ attorney, Allen proposed to his former son-in-law that “in consideration of his anticipated loss,” if Mr. Philips agreed to the adoption, Allen would pay support to Mr. Philips. Those same papers allege that the two men agreed that the support would total $2.7 million, which in the late 1950’s was a substantial amount of money. They also agreed that that amount would be paid out in annual increments of $30,000. Mr. Philips said that another stipulation of his agreement with Allen was that he cease any contact with his daughter and ex-wife. He added that Allen did not want his daughter to know about this agreement. The deal was sealed, he said, with a handshake, which is how Charles Allen often did business. Mr. Phillips said that Jacob Holtzmann, Allen’s attorney, told him that “shaking Charles Allen’s hand is better than shaking the hand of the President of the United States.”

Mr. Philips’ complaint indicates that the adoption took place in February 1959. He recalled the actual proceeding as “something out of Twilight Zone .” Mr. Philips claimed that the adoption took place in a conference room located in the offices of Mr. Holtzmann’s firm. He claimed that a partition had been erected to bisect the office, leaving enough room at one end to fit a desk and a chair. The judge handling the adoption sat there, enabling him to see Mr. Philips and Mrs. Kramer, who were on each side of the wall with their respective attorneys. Mr. Philips claimed that this was done so that his ex-wife did not have to see him. He said the proceeding lasted 10 minutes. “Then I left. I was very upset,” he said.

“You must be kidding me,” Mrs. Kramer said when she heard this account. According to her, the adoption had taken place, quite ordinarily, in “a place called Brooklyn. I had never in my life been to Brooklyn, O.K.? And I’m a sixth-generation New Yorker.”

Shortly after the adoption took place, Mr. Philips said that his mother, Beatrice Philips, succumbed to cancer. “My father accused me of killing my mother.” Mr. Philips said. He provided The Observer with a letter, dated Oct. 9, 1974, handwritten on what seemed to be a photocopy of Sherry-Netherland stationery that appeared to be from his father to James Philips’ wife at the time. “By now you must know how I feel. The adoption of Toni Philips has eliminated any possibility in the future of help from me. I have no intention of James ever being the recipient of my money.” The letter is signed, “H. Philips.”

Herman Philips died in June 1975. His will shows that he left virtually his entire estate to his second wife and to James Philips’ brother. In 1979, that estate was estimated to have a gross value of $1.8 million, but other sources familiar with the Philips family said that Mr. Philips probably lost out on an inheritance that would have been nearly 10 times that. Herman Philips did leave a $200,000 trust in James Philips’ name that threw off $10,000 a year. But a codicil to the will, filed in 1974, halved those amounts. Mr. Philips briefly contested the will, but eventually withdrew.

“He did keep his word,” Mr. Philips said of his father. “I’m not going to say that Charlie Allen kept his.”

Who’s Telling the Truth?

At this point, the question begs to be asked: Is James Philips lying? He rubbed his glasses, looked at his interviewer. “You know, there isn’t a soul alive, including my own family, that thinks I’m telling the truth about this. They all think that I got paid off or whatever.”

And there’s another question. Was Charles Allen simply doing yet another financial deal when he artfully arranged the adoption of his granddaughter and the payout scheme to his former son-in-law?

By the time Mr. Philips came into the Allens’ orbit, Charles and his brother were known for their golden guts when it came to investments. In the late 50’s, Charles did the deal that Forbes called his masterstroke: He paid $1 million for a controlling interest in a pharmaceutical company called Syntex Chemicals Inc. that would go on to develop the women’s oral contraceptive pill. In 1994, the company was sold for $5.3 billion.

Allen had long had ties to Hollywood. He was close to producer Ray Stark and had done a number of deals there. But Allen & Company’s association to the film industry would be cemented when Herbert’s son, Herbert A. Allen, put up $1 million of his own money to buy a controlling interest in Columbia Pictures in 1973. Young Herbert Allen, who is now not so young and is known widely for this annual summer media conferences in Sun Valley, Idaho, eventually sold Columbia Pictures to Coca-Cola for $40 million in cash and Coke stock that, as of 1994, was worth $250 million. Clearly, money was not the issue for the Allens.

According to the lawsuit, he “or his agents provided funds” to Mr. Philips for “the ensuing 33 years.” Mr. Philips said that between 1960 and the early 1970’s, his firm, Philips, Appel & Walden, “was doing really well” and that he was comfortably a millionaire. Indeed, Roger Coleman, a friend of Mr. Philips for some 30 years who also worked on Wall Street, said, “He was one of the best at picking stocks off the tape.” Mr. Coleman remembered another thing about his friend. “I know his checkered background,” he said. “I always thought that was unfortunate.”

“I had a lot of confidence in myself,” Mr. Philips said, and “it wasn’t important whether he gave me the money or not.” But around the bear market of the early 1970’s, Mr. Philips said that he began to run into trouble.

Around that time, he said, he sold his seat on the stock exchange and left his firm. In January 1972, the New York Stock Exchange took disciplinary action against him for, according to a memo to its members, “having engaged in conduct detrimental to the interest and welfare of the Exchange in that he conducted his personal financial affairs in a manner which adversely reflect on his financial integrity.” The memo said that he “would not be approved as a member or allied member and not be employed in any capacity with any member or member organization of the Exchange for five years.” Mr. Philips said that the penalty was a result of a relative who worked at his firm and ran up large amounts on his personal charge accounts around the city.

It was around this time that he and Toni re-established contact. Mr. Philips’ current wife at the time had a daughter in the same class as Toni at the Hewitt School. Mr. Philips said that Toni sent a note through a friend inviting him to see her at a school pageant. That night in the audience were both Mr. Philips and Mrs. Kramer. Mr. Philips said that he found his daughter in an upstairs classroom. “She threw her arms around me,” he said. “She was crying and upset. Her hands were cold.”

In another instance, the two arranged to have lunch at the Waldorf-Astoria hotel, where, he said, his daughter arrived with a male companion. “She wanted to know basically why all these things had happened,” Mr. Philips said. But, he added, “mainly because the boy was there, she was all bravado. She didn’t want to show her emotions.”

Mr. Philips said he eventually “realized there was never any way I could have a relationship with her.”

Although others familiar with the Hewitt School incident backed up Mr. Philips account, Toni (who is now 41, and spoke on condition that her new last name not be used) called it “an absolute lie.” She added: “He showed up at my school. I never invited him anywhere.” As for the Waldorf lunch, she claimed that her father initiated the meeting when she was about 16. “I thought I would meet him for myself, make up my own mind,” she said. “I found him an unsavory character. Not what you fantasized your long-lost father would be like.”

At one point during the conversation, Toni said that “my grandfather was sending [Mr. Philips] money just to keep him away from me,” which she likened to “blackmail.” But when this was raised later in the interview, she denied having said it. “I don’t know if there ever was a transaction,” she said. “I certainly hope not.”

In April 1974, Mr. Philips said that he had run out of patience with Charles Allen. “He made me go after every penny,” he said. Mr. Philips fired off an emotional handwritten six-page letter to his former father-in-law (which he provided to The Observer ). He wanted the balance of his $2.7 million. “I am no longer willing to continue this unbearable life for myself and my family because of your lack of interest in meeting your commitment,” he wrote. “I have protected Toni, Terry, Kramer and the Allens for the past 15 years.… I had to force myself and sacrifice everything, even my feelings for my own daughter. I can just imagine what was told to Toni over the years about her dad. I am positive the full truth wasn’t afforded to her.…”

“In conclusion,” Mr. Philips wrote, “my personal position is that of a ruined man. My wife is with child and my personal health is failing. There is no way I can reconstruct my life in a meaningful way, as was promised to me. Because of what has been and understood to be the truth, so help me God, I am totally unwilling to continue to be passive, permitting unreasonable suffering for my wife and children.”

After he sent the letter, Mr. Philips said he was called into the offices of Allen & Company. He said that Howard Holtzmann, Jacob Holtzmann’s son, led him to a conference room and opened the door. There, he claimed, sat Terry Allen, Irwin Kramer, his daughter Toni, Charles Allen and a number of other people whom he did not recognize. In front of each person, he said, sat a copy of his letter. “I was stunned,” Mr. Philips said. He recalled that Mrs. Kramer spoke first, and told him that he had “violated” the agreement by seeing Toni. Mr. Philips said he pointed out that his daughter had sought him out.

(Mrs. Kramer said, “There was a meeting” at Allen & Company, “where he was told by his daughter, ‘Leave my family alone,'” but she remembered Toni being younger. However, she said: “There was no letter.”)

After the meeting, Mr. Philips said that Charles Allen told him that he would not pay him a lump sum. Instead, he assigned a lawyer named James Deer to act as the liaison between him and his former son-in-law and to set up a payment schedule. (Deer submitted a sworn affidavit saying that he had been appointed to handle matters between Allen and Mr. Philips, “specifically, to handle the payment of monies from the former to the latter, based on an agreement made by the parties to induce Mr. Philips to maintain a friendly understanding with Charles Allen Jr., his former father-in-law.” But Deer has been dead for almost a decade, however, and is not around to elaborate on his statement.)

A few weeks later, Mr. Philips wrote another letter to Allen (which he provided to The Observer ) to say that he was “still in a state of shock” over the conference room scene. He also balked at a $2,000-a-month payment schedule that apparently had been decided upon. He closed by writing: “To the best of my knowledge Terry knows nothing of whats [sic] transpired between you & I. I know how much that means to you.”

He never got to ply that leverage, however, because not long after, Mr. Philips found himself in the weakest position of his life. In September 1974, Mr. Philips was indicted and in February 1975 pled guilty to the count of conspiracy to pass, possess and deal in counterfeit U.S. obligations, namely $5 million in Treasury bills. He was fined and sentenced to six months in prison. “Twenty-five years ago I did something wrong. I paid for it and went to jail for 60 days,” said Mr. Philips. “I got swept up in something with bad people, without knowing what I was getting involved with. When it became apparent, I cooperated with the authorities. This unfortunate incident has haunted me all my life, especially for the pain it had caused my family and friends.”

Letters From Wagner, Cohn

As the 1970’s segued into the 80’s, it appears that other prominent New Yorkers attempted to keep the peace between Allen and Mr. Philips. Mr. Philips showed the Observer a letter dated Dec. 18, 1979, from former mayor Robert Wagner. “I have spoken with Charles Allen on two occasions since our meeting two weeks ago,” wrote Mr. Wagner on stationery bearing the letterhead of his law firm, Finley, Kumble, Wagner, Heine & Underberg. “Mr. Allen has made it quite clear that he has ‘not neglected you,’ but because of difficulties within the family, he chooses to handle this matter in his own way.

“I can only give you this advice,” Mr. Wagner concluded. “Charles Allen has given his word to you; be assured that he will keep it. Do not upset the apple cart by attempting any legal measures at this time.”

Nearly a year later, Mr. Philips got a letter dated Sept. 8, 1980, from the attorney Roy Cohn. “Please understand that I do not represent you or Charles Allen [Mr. Cohn did, however, represent Mrs. Charles Allen],” Mr. Cohn wrote. “I have spoken with Charles Allen and he is very concerned that you would consider any kind of litigation. He has made it clear to me that he has every intention of meeting his obligation to you–having fully explored the situation I must side with him as you both agreed to keep your arrangement a private matter.”

In 1991, Mr. Philips said that he went into the hospital for a heart-bypass surgery, and from that point on, he said he never heard from Charles Allen again.

His lawsuit, which was filed in July 1993, lay dormant in New York State Supreme Court until September of last year, when his lawyer, Robert Graubard, succeeded in getting Allen’s estate listed as the defendant. Mr. Graubard declined to explain why the case had languished. But he did say that “What you have is a pattern of 20 years of Mr. Allen complying with an obligation which was acknowledged to a number of people.” Mr. Philips, he said, was suing “to enforce the terms of that obligation.” Mrs. Kramer’s attorney, Peter Bronstein, called Mr. Philips’ complaint “a bullshit lawsuit” that has no merit.

Reminders of a Former Life

A half-hour’s drive from Mr. Philips’ modest, shingled East Hampton home stands a reminder of the life he might have had. There, on exclusive Murray Lane, bounded by the Atlantic and a freshwater pond, sits the Southampton home of Mr. and Mrs. Kramer. Asked if he ever thinks about how his life might have played out if his marriage to Ms. Allen had survived, Mr. Philips replied, “Never.” He is currently divorced from his fourth wife (his third marriage, to a woman he had met had met on his honeymoon with Terry, was annulled). In addition to Toni, he has three daughters and currently lives with one of them in East Hampton.

There was a time when Mr. Philips wore custom-made Dunhill suits and shirts. Today, he is wearing sweat pants and a East Hampton Star T-shirt. “You don’t even recognize the person today,” said one of his longtime friends. “I think he’s just horrified at all of the things that happened in his life.” He still plays the market to make some income. He also spends some time wondering when there will be some movement in his case.

A few months ago, Mr. Philips said he called up Howard Holtzmann, who is listed as an executor of Charles Allen’s estate, to tell him that the issue had been dragging on. He said he suggested that both parties sit down and try to “get this thing worked out.” He said that Mr. Holtzmann eventually got back to him with a message from Mrs. Kramer. A message that, when The Observer asked, she vehemently denied sending. The exact wording of message, according to Mr. Philips? “Now we’re even-steven.”

Haunted by Giving Up His Baby, Philips Sues Charles Allen Family