Here's the summary of the WND article "Feds pay $5.8 billion to jail criminal aliens":

- Criminal aliens in federal prisons: 42,000 at end of 2001; 49,000 at end of 2004
- Criminal aliens in state prisons: 74,000 in FY2003 (80% of them in AZ, CA, FL, NY, and TX)
- Criminal aliens in local facilities: at least 147,000 in FY2003 (30% in five counties in AZ, SoCal, NY, and TX)
- The majority of the criminal aliens were Mexican citizens

The price tag? Around $2 billion per year over the past three years. Note that most of that amount was spent on the federal system, with a smaller portion going to partially reimburse states and local governments. Those reimbursements occur under the SCAAP ("State Criminal Alien Assistance Program") program. Since states and localities are only partially reimbursed, the full price tag for incarcerating those criminal aliens was probably a good bit higher.

WND gets their figures from the GAO, and you can read the GAO's report in this PDF file.

Note also this:

The Senate Judiciary Committee, on Thursday, March 17 [2005], approved Sen. Dianne Feinstein's bill to increase funding for the State Criminal Alien Assistance Program (SCAAP). SCAAP partially reimburses state and local governments for the costs of incarcerating undocumented criminal immigrants. Senator Barbara Boxer is a co-sponsor of the bill along with 10 other Senators from both parties...

...Last year's FY05 omnibus appropriations bill funded the program at $297 million, down from a high of $585 million several years ago. The Bush Administration has proposed zero-funding SCAAP in its last two budgets. California receives about 40 percent of SCAAP funding - almost $112 million of the $281.6 million total available in FY04.

UPDATE: "U.S. falls short on share of jail cost" has the California-specific details: "Illegal immigrants cost $635 million to lock up in '03, but state got $77 million..."