Warren Buffett, the iconic investor who leads Berkshire Hathaway (NYSE:BRK.A, NYSE:BRK.B), wrote in his annual letter to shareholders that he and Berkshire’s board are “enthusiastic” about the prospect of a new CEO to take the reins of the company. At 81, Buffett is painfully aware of his mortality.

And after lagging the market recently, investors are painfully aware that Berkshire hasn’t delivered quite the returns that it used to.

That begs the question: Are the best days of Buffett long behind him? And if so, does that mean Berkshire Hathaway is doomed to struggle without his steady hand at the helm?

Berkshire is one of the biggest success stories on Wall Street, no doubt about it. As an investment, it has delivered a compounded annual gain of 19.8% since 1964 — easily double the S&P 500’s 9.2% compounded return. More dramatically, Berkshire is up over 513,000% since 1965 vs. 6,400% for the S&P 500.

The last few years, however, haven’t been all that impressive.

• 2009: returns of 19.8% for Berkshire vs. 26.5% for the S&P 500
• 2010: returns of 13% for Berkshire vs. 15.1% for the S&P 500
• 2011: Returns of 4.6% vs. 2.1% for the S&P 500
• Total returns since 2009: Berkshire has added 20% in just over three years, while the S&P 500 is up 46.6%

Maybe a changing of the guard is in order.

Berkshire Earnings and Buffett’s Annual Report

Here are the earnings details from the annual report: For the fourth quarter, Berkshire’s net income declined to $3.05 billion from $4.38 billion a year earlier. That’s a 30% slide, blamed largely on derivatives bets from 2010 that juiced the earlier numbers. The company made $2 billion on the trades in Q4 last year, but has returned to the conventional Berkshire business of big investments and acquisitions as of late.

What’s more, as the profits dropped from derivatives, the losses from credit-default contracts rose. These are investments where Buffett and Berkshire bet on the ability (or inability) of borrowers to repay debt. Losses from credit-default contracts widened to $216 million from $157 million a year earlier.

On the plus side, Berkshire’s cash on hand surged to a mammoth $37.3 billion — adding $3 billion bucks in just three months. Unlike companies such as Apple (NASDAQ:AAPL) that notoriously hoard cash with no intention of acquisitions or dividends, Buffett and Berkshire actually tap this war chest for business moves — such as the $9 billion it used to buy Lubrizol in 2011, or recent equity stakes taken in tech and media companies like Liberty Media (NASDAQ:LMCA) and International Business Machines (NYSE:IBM).

Book value for Berkshire, a measure of assets minus liabilities, rose almost $5 billion from Q3 to Q4, totaling $164.9 billion from $160 billion.

But the headline number is, of course, the earnings. That’s because not only did Q4 profits slip, but full-year net income was also down about 30%, to $10.3 billion from $13 billion in fiscal 2010.

In short, there are a lot of positives. But profits — for Berkshire Hathaway or for its shareholders — just aren’t there.

Buffett and Berkshire Send Message of Patience

Here’s where things get interesting. The text of Buffett’s letter seems to acknowledge the inevitability of change — a change that may be coming sooner rather than later.

“The primary job of a Board of Directors is to see that the right people are running the business and to be sure that the next generation of leaders is identified and ready to take over tomorrow,” Buffett wrote. “As 2011 started, Todd Combs joined us as an investment manager, and shortly after year end Ted Weschler came aboard. Both of these men have outstanding investment skills and a deep commitment to Berkshire. Each will be handling a few billion dollars in 2012, but they have the brains, judgment and character to manage our entire portfolio when Charlie (Munger) and I are no longer running Berkshire.”

Combs has already left his mark with recent tech and media additions to the Berkshire portfolio, which in addition to Liberty Media and IBM also include DirecTV (NASDAQ:DTV) and Intel (NASDAQ:INTC). Those investments are definitely not in the same vein as mainstays of the consumer staple business and banking industry, which Buffett always gravitated to in years past.

As for the head honcho, however, that’s still a mystery. “Your Board is equally enthusiastic about my successor as CEO,” Buffett writes — without revealing the name. But the letter goes on to make it clear that everyone at Berkshire has great faith in the future of the company’s management beyond Buffett and his partner in crime, Charlie Munger.

But it’s interesting that at the same time, Buffett stresses the long-term nature of Berkshire’s investments in a way that almost urges patience.

“We view [Berkshire’s] holdings as partnership interests in wonderful businesses, not as marketable securities to be bought or sold based on their near-term prospects,” he says — perhaps to deflect concerns about near-term performance after some high-profile moves that included Lubrizol, the purchase of Burlington Northern Santa Fe railroad and a stake in preferred share of Bank of America (NYSE:BAC). Berkshire’s bargain buy of BofA included a 6% dividend to boot.

Buffett says the biggest positions that Berkshire holds, including almost 9% of soft-drink giant Coca-Cola (NYSE:KO), aren’t always reflected in Berkshire earnings because all BRK investors see is the profits via dividends paid out.

“Had we owned our present positions throughout last year, our dividends from the “Big Four” would have been $862 million. That’s all that would have been reported in Berkshire’s income statement. Our share of this quartet’s earnings, however, would have been far greater: $3.3 billion. Charlie and I believe that the $2.4 billion that goes unreported on our books creates at least that amount of value for Berkshire as it fuels earnings gains in future years.”

In short, the profits aren’t fully realized until Buffett sells its stakes — which could be years down the road.

This all sounds like an effort to prepare investors for immediate changes that may not have an immediate impact.

What’s more, investors should be disturbed at Buffett’s insistence that it takes years for profits to be fully realized. It has been four years since the panic of the financial crisis, but Buffett and Berkshire still continue to struggle in replicating prior successes.

If Berkshire investors need more patience despite three years of lagging returns, and if Berkshire’s strategy truly doesn’t materialize on the balance sheet or in share gains for years after moves are made, perhaps everyone better hope for change to come down sooner rather than later.

Investments like the Liberty Media purchase, a diversified 21st-century media company, or Berkshire’s new stake in small-cap dialysis provider DaVita (NYSE:DVA) show that the company is already trying to evolve. So, let’s just officially start the clock on Berkshire Hathaway 2.0, and jump right in.

After all, by Buffett’s and Berkshire’s admission, it’s going to take a few years to see if this plan succeeds. Unless the company wants to try investors’ patience — or unless it can guarantee a pop in the short-term — the company may be best served by just moving on.

Sacrilegious to say, I know. But maybe the best days of Warren Buffett and Berkshire Hathaway are indeed well behind us.

Jeff Reeves is the editor of InvestorPlace.com. Write him at editor@investorplace.com, follow him on Twitter via @JeffReevesIP and become a fan of InvestorPlace on Facebook. Jeff Reeves holds a position in Alcoa, but no other publicly traded stocks.