|
|
|
|
||
|
||
  |
Note to Researchers: Distribution of the document that follows is unrestricted. Original audio cassette(s), signed legal release forms, and edited interview transcripts are on file in the Defense Acquisition History Oral History Collection, U.S. Army Center of Military History, Fort Leslie J. McNair, Washington, D.C. Dr. David L. McNicol UNITED STATES ARMY DEFENSE ACQUISITION HISTORY PROJECT INTERVIEW OF DR. DAVID L. McNICOL CONDUCTED BY AT 15 SEPTEMBER 2004 Interviewee: Dr. David L. McNicol DR. HÖFIG: Actually, I'd like to start backwards, if I may, and ask you what you're doing now. DR. McNICOL: I'm a research staff member at the Institute for Defense Analyses (IDA). IDA is a Federally Funded Research and Development Center; its work is primarily for the Office of the Secretary of Defense (OSD) and the Joint Staff. I am in a division that does weapon system cost analyses, among other things, but I have done little work on costing since I joined IDA. I'm currently leading three projects. They're all for OSD. One of them has a cost estimation component to it; the others do not. The briefing on the History of the Cost Analysis Improvement Group (CAIG) (A copy of this briefing appears in the project file for this interview. ) is from an orientation course that IDA teaches, primarily for new staff members of OSD’s Office of Program Analysis and Evaluation. DR. HÖFIG: So continuing backwards, you were at the CAIG from 1988? DR. McNICOL: I started as chairman in February 1988 until September 30th, 2002. DR. HÖFIG: What is your own background? How did you come to that office? DR. McNICOL: I was in college and graduate school during a time when there was considerable interest in public service, and somewhere in that period my interest in government became a career choice. The only question for me was on what terms and by what path I’d go into government. My first four and one-half years out of graduate school were spent as an assistant professor in the department of Economics at the University of Pennsylvania. I started in government as a senior staff economist at the Council of Economic Advisors in January 1976. I liked the Council much better than I liked the academic world. I also liked Washington. When I joined the Council staff, I had a contract to go to the California Institute of Technology as a visiting associate professor of economics. I did go to CalTech, and enjoyed it very much, but I was looking for a job back in Washington from the day I arrived in Pasadena. I came back to work in the Treasury Department the following year, and stayed there for two and a half years. I then accepted a Senior Executive Service (SES) position in the Department of Energy (DOE), with an office that did modeling and forecasting of the energy markets. The head of the office was on leave from the University of California, Berkeley, where he was chairman of the Operations Research Department. As I recall, he heard about me through the academic grapevine. During my first ten months or so at DOE, I directed a large and very challenging study. The mission of the division was somewhat broadly defined. I joined PA&E in January1982, during the early stages of the Reagan buildup. The director of PA&E then was David Chu. (Dr. David S.C. Chu served as director and then assistant secretary of Defense (PA&E) from 1981 to 1993. In 2001, he was sworn in as under secretary of Defense for Personnel and Readiness.) David correctly anticipated that there was going to be substantial controversy over the economic consequences of the build-up. He also accurately foresaw that topic was one that the secretary and the deputy secretary of Defense would have to deal with. As director of PA&E, he would be the one principally providing support on the economic issues surrounding the defense buildup. So one of David’s early actions when he became director of PA&E in May 1981 was to establish an economics division. Reestablish it really, as there had been an economic group in PA&E and its predecessor office from the mid-1960s through the early 1970s. DR. HÖFIG: When you say the economic effects of defense spending, do you mean within the defense institution? DR. McNICOL: No, effects on the economy of the United States in terms of employment, inflation, interest rates, the international competitiveness of the US, shifts in the balance of payments, ownership of the U.S. debt, capital flows, and so on. Some of the work was interesting, but a much of it turned out to be simply firefighting. We wrote testimony. We helped with speeches and op-ed pieces and articles. We wrote talking points and memos and represented the OSD position at seminars and conferences and wrote more talking points. This was not a partisan issue. At that time, the Congress had a Democratic majority, but we were able to work smoothly with staff people on the Hill and, in particular, those with Congressman Aspin’s, (Les Aspin (D-WI) was chairman of the House Armed Services Committee from 1985 to 1987 and later secretary of Defense (1993-1994).) Armed Services Committee because they had the same feeling we did about the professional merits of the economic criticisms of the defense buildup that we were all hearing. Basically, our stance was: The defense buildup raises serious questions of international politics and the defense posture of the United States, but for heaven's sake, don't debate these questions in terms of silly assertions about the consequence for employment of a billion dollars of spending for defense and how much of it goes to Eastern Michigan. There was a group of people who were doing that, and the press seemed inclined to pay attention to them. DR. HÖFIG: Were more people accepting of the position that this is actually a big important deal that we have to get behind? DR. McNICOL: To the extent that I understood the situation, senior levels in both the executive branch and the Congress wanted the sorts of assertions we were reading in the newspapers—about the economic effects of defense spending—off the table. The added costs of defense clearly were relevant; they measure what else the country could buy if it chose to forgo the increased defense spending. It also was clearly fair to ask how the administration intended to adjust macroeconomic policy to accommodate the defense buildup. But the criticisms of the increased defense spending were not generally cast in those terms, and the professionals I spoke with—in DoD, OMB, the Council of Economic Advisors, Treasury, CBO, the Hill—thought that most of the issues then being raised about the economic effects of the defense buildup were bogus. You can get a quick sense of what an economist might say about the question by asking how the buildup would be financed. By an increase in the federal deficit? If so, will the added debt be accommodated by printing more money? Or will it be accommodated by the government bidding up interest rates, thereby displacing private investment? By tax increases? Or is the increase in defense spending to be accommodated by decreasing other government spending? Wouldn’t the economic effects of defense spending be nil if it simply displaces other federal spending? Why is NASA spending different from DoD or DoD different from NASA? The point is that economic effects of any increase in defense spending depend on what you do to finance it. So you really have two policy decisions. One, does the country wish to increase defense spending? And two, if yes, by what means do you wish to fund it? Some of the discussion of the economic issues within DoD provide a bit of comic relief. We had a conservative administration that had run against the kind of economic policies that were characteristic of liberal Democrats of previous administrations. Yet some people in DoD, both civilians and military—including some political appointees—wanted to write speeches on the economics of the defense buildup that were just a simple-minded version of Keynes. They did not realize that, however. They knew Keynesian economics was something the administration was strongly against, but that is about all they knew about it. The arguments they wanted to offer were drawn straight from a naïve version of Keynes that Keynes himself certainly would not have accepted. These were people who had no idea what the economic contours of the issues were. On a more concrete plane, we also pointed out to people in DoD who favored trying to sell the buildup on economic grounds that any attempt to do so was likely to draw very sharp and effective rebuttals from well-respected economists of both political persuasions. I was in that position for six years. The “economic effects of defense spending” issue faded during 1984-1985, in part due to our efforts. I would have left PA&E had it gone on. But the division grew and we got into other things, so I stayed around in that position until 1988—six years, almost to the day. DR. HÖFIG: And then you took over as chairman of the CAIG? DR. McNICOL: Formally as Deputy Director, PA&E, for Resource Analysis. Several positions came with the title Deputy Director (Resource Analysis), one of which was chairman of the CAIG. I was also ex officio recorder (that is, note-taker) of the Defense Resources Board and executive secretary of the Program Review Group. In these later positions, I worked on programming; the CAIG is part of the acquisition process at the OSD level. The weapon system costing work at that point was about 40 percent of the total workload of the office, and other work was about 60 percent. So I anticipated doing primarily what PA&E called programming work—working on what most of those outside DoD would call budgetary issues. Programming is the step between planning and budgeting in the Planning, Programming, and Budgeting System (PPBS). It's the part of the process that PA&E administered and was most active in. I hadn't intended to make weapon system costing a primary occupation. It was not long after I took over that the world changed a great deal. I began as CAIG chairman at the tag end of the second Reagan Administration. George H.W. Bush won the election in November 1988. As I recall, the Berlin Wall came down in November 1989, and the Soviet Union collapsed about eighteen months later. I remember vividly sitting up in the office of the Director of PA&E, literally looking out the window, as we contemplated what the consequences of the fall of the Berlin Wall were going to be and, in particular, how they would ripple into the part of the pond in which we swam. Secretary Cheney, (Dick Cheney served as secretary of Defense from 1989 to 1993.) reportedly was pointedly asking all the senior people in the military departments: “The world has changed. How has your service changed in response?” There was some very heavy lifting to be done during this period with the defense program. The Reagan buildup peaked with the fiscal year (FY) 1985 budget. What's not widely understood is that most of the cuts during '86-‘87-‘88 were of anticipated growth. So it wasn't that the services’ budgets went down much in comparison to what they had been recently; rather, they went flat, more or less. There was some pain attached to that because people had anticipated budget increases. The real pain began when Secretary Cheney came in and budgets went down. The procurement budget fell by about 60 percent. There were major programs cancelled, other major programs were cut in half. During his first year, Secretary Cheney used the programming process to make major cuts. But for reasons that I never knew the secretary and/or the deputy secretary Mr. Atwood and/or General Colin Powell, who was then chairman of the Joint Chiefs of Staff, became disenchanted with the programming process as a way of making the programmatic and budgetary changes that they sought. After the first year, those changes were largely made outside the programming process. There were other processes developed—more accurately, extemporized—to do this. Some of them you've probably heard of, for example, the process that General Powell led that resulted in the Base Force. And there was the Major Aircraft Review followed by the Major Ship Review and the Major Rotary Wing Aircraft Review. Later we had Defense Management Review. Let me bring this back to your question about the shape of the work I did as PA&E deputy for resource analysis. The programming work declined significantly as Secretary Cheney shifted the emphasis on resource issues to processes other than the programming phase of the PPBS. The programming function largely went into hibernation. There was one year in which OSD didn't even have a program review. While the programming work went down, the volume of CAIG work went up. The direct cause of this lay with the Inspector General (IG) review of the independent cost function, completed in early 1992. I wrote about it in my monograph on growth in acquisition costs. (David L. McNicol, Cost Growth in Major Weapon Procurement Programs, (Alexandria, VA: the Institute for Defense Analyses, 2004), see 46-47.) It was also mentioned, I think, in the CAIG history. (Donald Srull, The Cost Analysis Improvement Group: A History, (McLean, VA: Logistics Management Institute, 1998).) That report resulted in basic changes. Prior to the IG’s report, the CAIG did reviews of cost estimates made in the services. As a result of the IG’s recommendations, we switched from doing reviews of cost estimates to making our own estimates. Consequently, the CAIG work came to absorb a much larger share of my attention. DR. HÖFIG: Who is your model CAIG analyst? DR. McNICOL: Well, if I had to have just one—and one is not the right answer—I would find somebody who was trained as an engineer and had worked as an engineer. Weapon system cost estimation is about taking a description of a system that does not yet exist and the program that will be used to develop and build it and, based on that description, producing a cost estimate. Someone with a reasonable understanding of what goes on in development and manufacturing world is much more likely to produce a sensible cost estimate. That statement requires some qualification and extensions. There are cost estimating issues for which somebody with an economics background is most suitable; for example, those involving competition, financial consideration, or business-base issues. There are also some issues for which you’d like to have someone with an appreciation of contracting. Occasionally cost estimation involves fairly substantial modeling tasks, and for those, someone who's got an operations research background is helpful. So our policy was that the main-line CAIG analyst should be an engineer by training, but you've also got to have on the CAIG staff people with other sorts of backgrounds. DR. HÖFIG: Can you tell me how a review works? Where does it start? DR. McNICOL: There are two relatively straightforward ways to answer the question. One is in terms of what was supposed to happen and the other in terms of what usually did happen. I’ll first sketch what was supposed to happen. The processes laid out in these documents varied from time to time. There was a substantial change around 1987 associated with the Packard Commission report and the Goldwater-Nichols Act. There was another around 1989-1990 when Don Yockey was the USD(A) (Donald J. Yockey was under secretary of Defense (Acquisition) from 1991 to 1993. He had been acting under secretary for the first half of 1991; prior to that he served as principal deputy under secretary for Acquisition in 1990.) and John Christie was director of what was then called the Office of Acquisition Program Integration. I don't think it's a good idea for me to attempt a description of the specifics of what the process was and the various changes made to it, but let me try to give you the view from 30,000 feet of the pre-IG report independent costing process. The triggering event for a CAIG cost estimate was ordinarily the scheduling of a Defense Acquisition Board (DAB) meeting. That would typically happen three or four months before the meeting. Once we knew when the DAB was to be, a member of the CAIG staff would be assigned to the case. That person would consult with their service counterparts and schedule a CAIG meeting. There were usually two service cost estimates prepared: a program office estimate (POE); and another estimate that came to be called a Component Cost Analysis (CCA), which was done by a headquarters-level unit in the service: The Air Force CCAs were prepared by the Air Force Cost Analysis Agency; the Navy’s by the Navy Center for Cost Analysis; and the Army’s by the Cost and Economic Analysis Center. At the CAIG meeting, the program manager or someone acting for him, would brief the POE, which was typically also the estimate adopted by the sponsoring service. The CCA would be briefed, by someone from the cost center that had prepared the estimate. The service was supposed to provide the full documentation of the estimates not long after the CAIG meeting. In its narrowest terms, the job of the CAIG was to review the POE and CCA estimates and say, “Okay, that part of the estimate is sour. It must be replaced for the following reasons. A better estimate is $X billions.” As I recall, under the rules in place in the late 1980s, the POE and the CCA were supposed to be provided to the CAIG in draft form ninety days before the DAB and in final form a month before the DAB. I’m hazy on this because we rarely saw even a draft cost estimate much more than a month before the DAB. We made a trade in the revised rules issued when Don Yockey was USD(A). In the revised rules, the draft cost estimates were not due until forty-five days before the DAB committee meeting (which ordinarily was two weeks before the DAB), but that deadline would be enforced, and in fact it was. We tried to schedule the CAIG meeting about a month before the DAB. The nominal purpose of the meeting was to give the CAIG chairman and other members of the CAIG a chance to ask questions about the estimate and to probe parts of it that they thought were weak or dubious. (The CAIG was then formally an intra-OSD committee.) Its real purpose, however, was to provide some assurance that the program brought to the DAB for milestone approval would be reasonably completely specified, and that the service and the USD(A) would have some idea of what it was likely to cost. Program managers, for understandable reasons, typically like to leave options open as long as they can. USD(A)s and other senior people in OSD like to be told just what it is that they are being asked to approve. That is where costing comes in: To the extent that a program has substantial ambiguities, it cannot be costed. Consequently, costing tends to identify the major programmatic issues that remain to be resolved, and to force decisions on them. My sense it that the senior people in OSD during the late 1980s to the mid-1990s tended to value at least as much the role of the CAIG in gaining program definition as it did the CAIG estimates as such. DR. HÖFIG: But the material was coming from the services? DR. McNICOL: The estimates came from the services for CAIG review. Let me give you a simple example of one sort of thing the CAIG looked for. The unit costs of a system go down as the cumulative quantity produced increases. The rate at which those costs go down as cumulative output increases is the slope of the learning curve, which is sometimes called instead the cost improvement curve. The value used for the slope in the estimate is important; differences of two or three percentage points in slope can easily be billions of dollars. A classic CAIG criticism of a service cost estimate would be along the following lines: “Your estimate for the direct manufacturing labor costs on this tactical aircraft assumes a slope of 72 percent. The steepest slope for direct manufacturing labor ever observed post-World War II on a tactical aircraft program is about 78 percent. Your assumption is untenable. A more realistic assumption would be the median slope on tactical aircraft, around 82 percent.” So the CAIG would write a memo that says assumptions on reductions of unit costs in manufacturing are much more optimistic than is warranted by the facts of the program and past history. The memo also would (or should) indicate what a more realistic assumption would be, and what that assumption implied for estimated cost. In fact, the people who did cost estimation for the services were competent and, after the early days of the CAIG process, rarely made mistakes like that. Consequently, most of the issues raised by the CAIG would be much more tangled. For example, the program office might point to a set of cost reduction initiatives to justify use of a slope much steeper than the historical average. Then the problem would be to evaluate those initiatives. DR. HÖFIG: But the base information is still what the services came up with? DR. McNICOL: Yes, in two respects. First, the description of the program itself—what it is you're costing—is what the services propose to the USD(A). Second, until about 1993, when the effects of the IG report were felt, the CAIG took as its point of departure cost estimates provided by the service. (The CAIG did have its own sources of costing data and our own information on learning curve slopes and cost estimating relationships and other cost estimating tools.) That was the nominal old process. I should stress, again, that the process frequently didn't work as DoDD 5001/DoDI 5000.2 said it should. We constantly had problems of timelines being truncated. Documentation frequently was weeks or months late, or it was seriously incomplete, or it was incoherent. What was the CAIG’s recourse at that point? I could call some person in the service, and say that we're really very unhappy with the fact that we had not gotten documentation, and they'd say, “Oh, that's terrible. We'll see what we can do.” Or I could go the director of PA&E to get his help, or to the USD(A). I could easily spend several days waiting to get on the USD(A)’s calendar. I generally could get to the director of PA&E quickly, but the services’ procurement bureaucracies, by and large, had no particular pattern of established business with PA&E and so they might not be very responsive. To be blunt about it, we were easy to stiff. Getting access to program data also was frequently very difficult. To do a good estimate, the CAIG action officer needed some access to program office staff. The CAIG also probably needs to visit the contractors and the principal subcontractors. Arranging that sort of thing could be time-consuming, and we typically did not have much time. So it was a very rickety process. DR. HÖFIG: How could you fix it? DR. McNICOL: Not everybody agreed that it was broken. It had a cop walking the beat and, a cop walking the beat tends to reduce the amount of street crime. That was more or less what the CAIG was doing. The ability of the services to put forward unrealistically optimistic estimates was considerably reduced by this process, even though the CAIG was mainly reacting to what the services brought forward. Hence, directors of PA&E did not see the CAIG as a high priority for any additional resources. The director of PA&E expected me to maintain a capability for doing independent weapons systems costing in those cases in which the secretary or deputy secretary had an important decision to make that required a realistic cost estimate. This was a classic OSD stance: “We will set the policies; we will review the service proposal; we will complain when the services go awry; and in major cases, we will find a way to make our complaint stick.” The programs in question are those with which the senior leadership—secretary, deputy secretary, under secretary for Acquisition—have reasons to become substantially engaged. I suppose the classic instance would have been the Strategic Defense Initiative (SDIO) in '88-‘89. The cost estimates that were being tossed around were widely regarded as nothing more than guesses, probably highly optimistic guesses. The senior leadership of the department wanted an estimate that reflected some oversight by careful people with no agenda. Incidentally, attempting to make an accurate estimate for a system at the stage of development SDIO was then at is heroic. What you can do is to say, “Okay, this is what we assume is going to be developed and procured, and this is about what a program with that content is going to cost.” That is, the cost estimators need to lay out what program assumptions the cost estimate reflects. The senior people who look at the cost estimate aren’t gullible; given an estimate described as a floor resting on very preliminary program assumptions, they have a fair idea of what costs would prove to be if the program is pursued. I should add that a cop walking the beat often was not enough to satisfy the USD(A), however. The USD(A) had a more demanding role for cost analysis and tended to have a broader view of its utility. So there was usually latent tension between PA&E and the office of the under secretary of Defense for Acquisition on how to structure weapon system costing at the OSD level. DR. HÖFIG: That's interesting because cost estimation lives in PA&E. DR. McNICOL: It does, and there are some historical reasons for that. You may be interested in the story of how it came to be that the CAIG is in PA&E. It goes back to the establishment of the Planning, Programming, and Budgeting System (PPBS) by Robert McNamara. He became secretary of Defense in 1961. Apparently before he arrived, Mr. McNamara had decided that he was going to install the PPBS. He brought with him Charles Hitch. (Charles J. Hitch served as assistant secretary of Defense (Comptroller), 1961-1965. At the time, the Systems Analysis function belonged to the comptroller’s office.) PPBS had evolved at RAND in the 1950s. Hitch was responsible for implementing PPBS. The group at RAND doing PPBS had working with them a group of weapons systems costers, and Charles Hitch had installed part of that group in Washington. The Office of Systems Analysis (OSA) along with another new group in Comptroller that included the weapon system costers implemented PPBS. To say why the OSA needed weapon system costers, I need to say a little about the PPBS in the McNamara years. During those early years, the central elements of the Five-Year Defense Program—now the Future-Years Defense Program (FYDP)—were built in OSD. In particular, the major procurement and the major R&D programs in the FYDP reflected secretary of Defense (or presidential) decisions made through a process orchestrated—in fact, largely dominated—by OSA. (The services filled out the remainder of the FYDP—operations and maintenance, military construction, family housing, and so on.) OSA consequently required some serious weapon systems costing capability to develop the options on specific procurement programs that they took to the secretary of Defense The CAIG was created in 1972. At that point, PA&E had the only cost shop in OSD. I believe that, more than any other factor, was the reason the CAIG was located in PA&E. During the thirty years that the CAIG has been in PA&E, PA&E has drifted in the direction of not having a great deal of use for a capable weapon system cost group, but AT&L has drifted toward using cost analysis more. That would argue for moving the CAIG to AT&L. Still, there are some reasons for leaving the CAIG in PA&E. Being located in PA&E helps with maintaining independence. Having the CAIG chairman at the level of a PA&E deputy helps the CAIG continue to work for the senior DoD leadership rather than the Congress (because PA&E deputies do not testify). Also, PA&E’s reputation and the career path it offered perhaps made it easier to attract exceptionally qualified staff. We worked through all of this in late 1991 in connection with the DoD IG’s review of the independent costing function in DoD. There was then some active consideration of moving the CAIG out of PA&E. I opposed that at the time, but in retrospect, I’m not certain that I was correct to do so. DR. HÖFIG: Now, does the “independence” in the group’s name mean the CAIG is independent of the services? DR. McNICOL: Yes. Statute requires that the milestone decision authority consider an independent cost estimate at certain decision points. Fundamentally that means an estimate made by a unit that does not report up the same chain of command—typically a service chain of command—as the manager of the program. But there is more to independence than the statutory provisions, because, like people in the services, PA&E staff and others in OSD can have strong preferences on various programs and may have invested in those positions over a period of years. The CAIG chairman has to make sure that his analysts don't pick up enthusiasms from PA&E and start pulling punches on programs that are currently favored, or using cost issues as tools for opposing programs that are not favored. DR. HÖFIG: What about political influence? I gather from your book that the CAIG had been called on to do some costing source selection alternatives. You were called to help identify sources for savings. There was also Secretary Cheney’s instruction to come up with program deletions. These are potentially political activities. DR. McNICOL: You're right; these activities can involve working pretty close to the horns. I never experienced overt political pressure. The people I worked for understood what the career civil service is all about. But having said that, these are matters of great importance to some people and the passions can be pretty fierce. Anyone not a candidate for sainthood who has strong feelings about a program is on a slippery slope to partisanship. This was a problem I thought about a good deal and had to think through again from time to time. It was necessary to make clear to the CAIG analysts that their job was to make a realistic cost estimate. It was also necessary for the CAIG to be as invisible as possible. For example, I would never go to a reception on the Hill. I did not cultivate any acquaintances employed on Capitol Hill. I did not testify and avoided going up to the Hill to meet with staff. I didn't take phone calls from the press. I would almost never take a meeting with weapon systems contractors. DR. HÖFIG: And the logic here is that you work for OSD? DR. McNICOL: You work for OSD. I used to point out to the CAIG analysts that we only had only three clients: the secretary, the deputy secretary, and the under secretary (AT&L). In order to preserve the CAIG’s role as an independent, in-house adviser to these people, you have to stay out of the partisan whirlpools on the Hill, and that requires staying invisible. Put differently, in order to avoid being ignored, you must conduct yourself so that you can be ignored. I don't think that most of the people in the Congress we came into contact with appreciated that the CAIG could not both serve them directly and remain an in-house DoD adviser. It was explained to them, but most of them just didn't get it. DR. HÖFIG: So if the under secretary for AT&L wanted to talk about these things, he's welcome to do it, but you didn't want to do it? DR. McNICOL: I would not volunteer. DR. HÖFIG: Congress got kind of excited about the CAIG a couple of times. DR. McNICOL: They did. You mean the A-12 imbroglio? The work we did for the A-12 in connection with the 1990 Major Aircraft Review (MAR) is a classic example of costing work done by Resource Analysis under the PA&E banner rather than the CAIG’s. I tried hard to reserve the CAIG label for estimates we did in the DAB process, and which were therefore done to a certain quality standard. The MAR was an ad hoc process, not the regular acquisition process; our work in the MAR was not done or using CAIG procedures; and it was not a CAIG estimate. Let me explain. I was in the office between Christmas and New Year’s 1988 trying to clear my desk, and I eventually worked my in-basket down to a memo signed by the Secretary of Defense. It announced that we were going to have a Major Aircraft Review and that the details would be forthcoming. A couple of days later, we got a memo signed by John Betti, who was USD(A). That memo restated the objectives of the MAR and said that it was to be completed by mid-February. As I recall, that memo also stated that the MAR would be outside the DAB process. At that point we knew the objectives of the MAR, but not how it would be structured. Some, including David Chu, had in mind what would have amounted to a DAB review for each of the three programs. That would have required my office to do three independent cost estimates (ICEs): one each for the C-17, B-2, and A-12. I went to David and said plainly that there was no way that we could do ICEs for the C-17, B-2, and A-12 by mid-February. (The deadline was later extended to the end of March as I recall, but that would not have made it possible to do the job.) This wasn’t a matter of willingness on my part to do the estimates; my point was just that we could not move our hands fast enough to get the job done by the end of March, much less by mid-February. David Chu seemed very reluctant to go to the new leadership and say this. But it didn't matter because the center of gravity on decision-making shifted away from David and the other people who had similar views on how the MAR should be structured. The MAR design settled on in early January had almost nothing in common with the kind of thing that goes on with the DAB. The senior leadership apparently didn't think the DAB process would work well for what they wanted to do. They set up the MAR as a large and very quick Cost and Operational Effectiveness Analysis (COEA). The MAR was designed to let the secretary decide how many of these aircraft we should buy and, in fact, whether we should be buying them at all. It had little in common with a DAB program review. We wound up looking at something like thirty-two platforms—thirty-two separate types of aircraft—and all together costed something over a hundred alternatives. The summary notebook was about four inches thick. Later, some people in the Congress took what we did on the A-12 for the MAR, held it up to the standards of what ought to be done in an independent cost estimate for the DAB, and found it wanting. I agree; what we did was not an independent cost estimate like we would do for a DAB. That isn't what we were directed to do, and isn’t what we did. Why the people on the Hill responded as they did is something that you'd have to ask them. But you are correct if your intent is to suggest that the A-12 had a major influence on the IG’s assessment of independent costing in DoD. In particular, some of the people on the Hill who criticized the CAIG’s work on the A-12 explicitly raised the question of whether the process then in place, which involved both the service cost centers and the CAIG, satisfied the statutory requirement for independent estimates. DR. HÖFIG: The statute requiring independent cost analysis? DR. McNICOL: Yes. The process DoD used at that time on its face did not conform to the statute. I think the career staff people on the Hill who were watching the DoD acquisition program pretty much knew how the CAIG process worked. My sense is that they thought the outcomes were within their tolerance, so they weren't upset. But when someone drops a penalty flag, you're going to go back and read the statute. That's what happened. We informally took this matter to the OSD General Counsel’s (GC) office. I asked the classic question: What conduct must I avoid in order to not generate evidence of an infraction of this statute? The GC lawyer we spoke with—I’m translating into blunter language his careful response—said two things. One, I'm pretty sure from reading the statute there would be a way to satisfy it with a process somewhat along the lines in use. Two, I can't tell you what that process would be. It's like pornography. I'll know it when I see it. This was not terribly helpful. I think the powers-that-be understood that the GC’s position left any CAIG chairman without a way to conduct the operations of the CAIG that was consistent with how the statute was being read. DR. HÖFIG: Until the IG's report came down? DR. McNICOL: Yes. As a result of the IG’s recommendations, CAIG staffing increased from about ten man-years of effort annually to well over thirty. We hired nineteen additional people for CAIG work, and staff time that had earlier gone elsewhere shifted into costing. With the added staff, we could just literally do what the statute directed. And in fact we began to do so. DR. HÖFIG: And it's still doing that? DR. McNICOL: I believe so, although I have not followed the CAIG’s work closely since I left. There is a complication with this otherwise straightforward point. Occasionally, the service will make a programmatic assumption that is unreasonably optimistic. For example, it might assume a four-year development schedule, while past precedents suggest that it is likely to take, say, six years. It is part of the CAIG’s job to point out such unrealistic assumptions to the best of its ability. But the CAIG was not set up to do design engineering work or to raise the possibility that some other program might be more cost effective. That is PA&E’s job; the CAIG’s is to estimate the cost of the proposed program. I think the IG and our critics on the Hill never quite understood the difference in the CAIG’s role in the DAB versus PA&E’s. They are distinct; the director of PA&E is a member of the DAB; the CAIG chairman, independently of his role as a PA&E deputy, attends DAB meetings as an adviser. DR. HÖFIG: I was interested to go through the CAIG history and the discussion of parametric costing. You say “historical costing data” and you think: how would you do that with something that's brand new? DR. McNICOL: I think the key point to notice is that once you go down to a sufficient level of detail, these new systems are not brand new. A weapon system cost estimate isn't a number; it is dozens or hundreds of numbers. I've heard Air Force officers characterize an aircraft as a collection of spare parts flying in formation. Thinking along these lines, you can start asking in a constructive way how different are the actuators from the actuators used in an aircraft already in service? What about the landing gear—how different are they? What are the changes you're making in the engine, and so on and so on and so on? Usually—not always, but usually—the data from the past can be scaled to take account of the differences in the new systems. But then you're working at a level of detail that is going to make most people's eyes glaze over. You cannot talk to congressmen and senators about that level of detail. DR. HÖFIG: How much of a corrective influence or an instructive influence does the CAIG analysis have on the process? DR. McNICOL: I think the accurate answer to your question is: I don't really know in a concerted, defensible way. I think it’s clear that the institution of parametric costing in around 1972 did have an effect. It made the estimates more realistic, and that's statistically demonstrable. So you can say that without the CAIG you would on average see less realistic cost estimates and more cost growth. There are indications that are not conclusive that the improvements in the CAIG have made the estimates more realistic. After that, what you have to go by is mainly anecdotal evidence. I tend to remember all of my successes and forget my failures, but certainly if you give me a few minutes, I could pull back into memory a lot of successes. They tend to be at a fairly again detailed level. There are only a very few cases that resulted in a hit-you-between-the-eyes result: They had a bad estimate; we gave them a better estimate; they adopted ours and the world was much better because of it. I have a few stories like that, but only a few. A more typical case is that you get a very good analyst involved at, say, six months before the DAB, and the analyst really learns the program and helps the program office out in lots of ways. You avoid troubles that would otherwise come up. You get more realistic test schedules. You don't forget to cost initial spares and on and on and on. DR. HÖFIG: Now, you wrote in the book that the relationship between the CAIG and the services, kind of these working parts, is "not as acrimonious as one might expect in this process." Is that true? DR. McNICOL: Generally. DR. HÖFIG: How did that happen? DR. McNICOL: I'm not sure there's a single answer. The one I'm inclined to give you is a little self-serving: I worked at making it true. But I think there are a variety of things that went on. One of the most important was the institution by Paul Kaminski, when he was under secretary of Defense for Acquisition and Technology during 1994-1997, of Integrated Product Teams (IPTs). I think the services came to accept working in IPTs. They also came to accept the need for a Cost Analysis Requirements Document (CARD) that lays out the program to be costed. With the IPTs and CARDs a lot of the sources of day-by-day bickering went away. Actually, I think the IPTs and the CARDs may have been more important than the CAIG starting to do its own estimates. Another thing to note is that you get bad relations between the CAIG and the services, by and large, only when a program manager for one reason or another has been issued an unrealistic cost estimate and/or an unrealistic schedule. If he's been issued these, there isn't a lot he can do about it, and from his perspective, the CAIG is working overtime, pounding on him about something he can’t change. Program managers naturally don't regard that as terribly constructive, and some of the interactions involved can get testy. I can understand the program manager’s perspective, but the CAIG is not there to make that program manager's life easier. It’s there to make sure that the USD(AT&L) is provided with a realistic cost estimate. As one retired officer put it to me, "The services understand why we have to have a CAIG. But you guys are cops, and nobody likes cops." That seems roughly right. DR. HÖFIG: So they won't go to lunch with you, but they accept you there. Only the function is not sufficiently accepted that the services are going to change a lot the way they do their own costing? DR. McNICOL: That's an interesting question that my book set out to answer. I'm not sure that the answers I got are entirely crisp, but to a first approximation, the Navy doesn't have a systemic problem with the realism of its estimates for major systems. Through my tenure, the Navy’s estimates tended to be pretty good. I understand that at least one of the Navy estimates that I worked on has gone sour since I left, but the Navy does not systematically under-cost programs by an unacceptable degree. DR. HÖFIG: Has it always costed so well? DR. McNICOL: No. The Navy apparently changed its practices in the '70s. The data I looked at suggest that that the Navy's costing was distinctly optimistic in the early 1970s and then became more realistic. The Air Force certainly knows how to cost programs, but the Air Force has, on particular instances, advanced some estimates that we thought were clearly not realistic. If the Army wants to, it can make a good cost estimate. It isn't clear to me that the problem the Army has with the cost growth is primarily with poor costing of programs brought forward to the DAB. My own view, which the data don't fully demonstrate, is that the Army has a tendency to take programs into engineering and manufacturing development too early. Then the Army changes the programs, and the result is substantial cost growth. DR. HÖFIG: Where you have, just as an example, these isolated cases of the Air Force with over-optimistic costing to this day, why did they persist? The services know that the CAIG is going to review these estimates. They know that the CAIG is going to come up with other numbers. Is this just remnant cultural habit? DR. McNICOL: No. Tolstoy applies here—all happy families the same, every unhappy family different. I think the reasons are probably different in each case, but it isn't cultural in any simple sense. Each of the services is much too careful to commit large blocks of its resources without some thought. I believe you'll find that the explanations of exceptionally large cost growth almost always has to do with budgetary positions, with force structure, with industrial base, something like that. DR. HÖFIG: So, there are still pressures on the services to do optimistic costing? DR. McNICOL: Absolutely. But I think the question isn't optimistic, it's how optimistic. The services and the contractors both have reasons to be somewhat optimistic in costing, but serious problems arise only in cases in which they get carried away with it. There is a certain amount of elasticity in the budgeting system and in the contracts themselves to absorb some degree of optimism. If a service starts costing major systems at a half or two-thirds what they ought to be costed at, however, it is out of the realm of the elasticity in the programs and the budget. Then bad things are going to happen. I know of some instances in which an unrealistic estimate was adopted just because of bad costing. We do sometimes just flat make mistakes, do things wrong, and inadvertently adopt a bad number. But my sense is that what in retrospect appear to be bad estimates at least a fair fraction of the time grow out of other decisions. So instead of making a cost estimate, a service lets other decisions it has made constrain it to an unreasonably optimistic cost estimate. DR. HÖFIG: The first Gulf War happened in 1991, just as you were kind of getting ramped up as chairman of the CAIG. Did the war change the activity at all? DR. McNICOL: It did change the activity in PA&E. Programming work in OSD dropped off during the Gulf War because there was no interest in either new initiatives, which would cost too much, or in cutting the services. So I look back on '91 as a relatively placid time professionally for me. '88-‘89-‘90 were much tougher years for me. Right out of the chutes, I had the B-2 estimate. I was sitting at lunch in the Executive Dining Room as the new CAIG chairman. The USD(A) used to sit there also, and the second or third time he mentioned that he was really quite concerned about the B-2 cost estimate and wondered what was going to happen, it dawned on me that maybe that was a hint that I was supposed to go do something about it. So we put together a team and estimated the cost of the B-2 program. At the same time, we had big drill going on in the Strategic Defense Initiative Organization Systems (SDIO) and the C‑17 Milestone IIIA decision in December 1988. We also had the predecessor of the F-22 earlier in 1988. In 1989, we also had a high level of CAIG activity and, in addition, 1989 was the year we had three program reviews. We had one in March when Secretary Cheney came in. We had the normal process during the summer and then, when the Berlin Wall came down, we had a third. In 1990, we had the Major Aircraft Review, Major Ship Review, and the Major Rotary Wing Aircraft Review. At the end of 1990 into 1991, we had the A-12. I think we started doing the A‑12 in October of '90. The workload was far above what it had been during the mid-1980s. It got to the point where we were not able to hire people we wanted to hire because my office had gotten the reputation as pressure cooker. I recall one naval officer who worked for us. By mid-April 1990, he had been in the office forty-three days straight, right through the weekends, and most of those were twelve- and fourteen-hour days. His division director and I went to this officer, and using the correct Navy language, told him he was to take the weekend off, and, if he decided not to come in Monday, that was fine, too. I had another experience emblematic of how we worked then. It was late in the workweek, Thursday or Friday, at 9:30 at night. I was walking out the door of the Pentagon to South Parking to get in my car to drive home. As I was going out, one of the CAIG analysts was walking in; he was coming back to work at 9:30 at night. And he managed a cheerful greeting. It was staff people like that who got us through these years. We started at 6:30 in the morning, and we would go through until 7:30, 8:30, 9:30 at night, sometimes later. Saturdays we generally didn’t start until 9 and tried to knock off by 5 or so; Sundays were noon to about 5. Just to keep up. The entire CAIG staff was working those hours. DR. HÖFIG: When you left the CAIG, you had a full staff? Or a fuller staff? DR. McNICOL: I left with fourteen vacancies. At that point, I was writing doomsday memos to the powers-that-be. I don't know if I actually said, “You're in violation of the agreement with the IG.” I did tell them that if they didn't fill those fourteen vacancies, they weren't going to have a CAIG anymore. I'd been complaining about those vacancies for months. They finally got the message and my successor moved very vigorously to fill the vacancies. I don't think the CAIG could have dodged the MDA work, but I told my bosses that we were not staffed to do it. I had done the staffing computations in 1991 in connection with the IG report. I think some of the new political appointees (although not Mr. Aldridge (Edward C. “Pete” Aldridge was under secretary of Defense (Acquisition, Technology, and Logistics) from 2001 to 2003. Earlier, he served as secretary of the Air Force, 1986-1989.)) thought that CAIG staffing had to be padded. They were unaware of the strong pressures I was under in 1991-1992 to keep the staffing increase down to a very minimal level. That staffing computation was not remotely padded. It would be instructive to compare the cost of the CAIG estimate to the cost of an estimate that you get from a private company. I would guess the full cost of a CAIG estimate is somewhere around 20 percent to 30 percent of what you'd pay to get an estimate done by a private company. Part of the difference is explained by the fact that the CAIG doesn't do the documentation the private company does. But it is also true that the CAIG is a lean operation. My understanding is they're very busy over there right now. DR. HÖFIG: What would you improve in the process? DR. McNICOL: The federal government hiring process is broken, and that really makes life difficult for CAIG chairmen. You can't hire the people you need. The hiring process takes too long. You don't have the right flexibility on pay. You can't go high enough on pay. The main opportunities I see for reducing cost growth rest with how DoD budgets for major systems. There are some changes in budgeting rules for major systems that could be made and that would substantially reduce the services’ temptation to adopt unreasonably optimistic estimates. I suppose the thing that's most troubling to me in retrospect has to do with where the boundaries of cost analysis are set. The sense in DoD generally seems to be that cost estimators exist mainly to provide numbers for the budget. The CAIG’s role in the DAB process, however, is more that of a diagnostics group. What the CAIG asks is: What's the health of this program? Is everything lashed down? Is this a program that's capable of executing in the way that it's laid out? The answers to these questions and others in the same vein may be articulated in terms of cost, but to get the answers the CAIG analysts must learn a lot about the test program, for example, and about whether the development program has been funded consistently with the planned staffing loads and so on. These questions amount to asking whether the pieces of the program are attached in ways that will bear the heavy loads placed on them in the normal course of the acquisition process. I think the utility of the CAIG’s work in this respect is not understood. (End of Interview of Dr. David L. McNicol.) * * * * * |
Project Home | Newsletters | Research | Endorsements | Links | Contact CMH Online page last updated 10 March 2004 |