Past Performance Pain
I recently submitted a proposal for a project that I am very excited about. The procurement is seeking to do a lot of the right things — using cloud platforms and open source languages in delivery, and weighing technical challenge performance heavily in the evaluation. However, the procurement still had flaws, including usage of past performance.
What is Past Performance?
Naturally, those selecting a services firm want to understand the quality of work for a firm. This is why, in daily life, we might speak to references when hiring a person for a job, or speak with past customers of a potential contractor for a home renovation. Contracting Officers have some typical tools to understand that a potential contractor does good work, and each have a specific meaning.
- Corporate Experience are case study write-ups provided by the proposing firm. They tell the story of how the contractor has performed similar work, and generally what the results were. It provides comfort to evaluators that this experience lowers risk in delivery of the upcoming work.
- Past Performance is more specific, designed to measure how well the firm meets its commitments. Many procurements ask for relevance (the scope of work should be similar), but not always. It is more about meeting quality, cost, and schedule commitments — a demonstration of reliability.
In other words, Corporate Experience is about what relevant work you’ve done, and Past Performance is about how well you have met commitments, regardless of the scope of work. These tools provide a check for evaluators, to make sure that the contractors can’t just hire a great proposal writer for a tech approach and be unprepared for actual delivery.
What’s wrong with past performance?
It’s natural to guard against inflated contractor claims. However, the information gained isn’t always of great value, and the process creates burden. Some of the general issues with these evaluations include:
- No guarantee: In financial products, we often see the phrase “Past performance is no guarantee of future results.” This is true in services too. How much does it really matter that work happened to be done by the same firm? Maybe a lot, maybe not. Will the same people be involved? Is it a different division/practice of a very large firm?
- Contractor spin: In any proposal document, vendors put their best foot forward. We highlight what is favorable or relevant to the given opportunity. There is nothing necessarily wrong with this, but you are getting a filtered and framed version of the truth.
- Cherry picking: The contractor generally chooses the reference person, so they will only supply references where there is a positive relationship. For small firms, this is less of an issue (because of a smaller pool to choose from). But if you ask for 3 recent references, a large firm could theoretically have delivered great on 3 of 100 projects, and been a complete disaster on the other 97. (Which reminds me of this xkcd comic, including the hovertext.)
- Reducing competition: Past performance questionnaires are a royal pain. It is a bid/no-bid decision factor for firms. You have to ask your customer to fill out a form, sometimes a tedious exercise (see next item). When you ask for past performance, you reduce the pool of competition.
- Government burden: In my recent case, government references had to print out a poorly constructed PDF file, write in answers, scan it in, then submit by email. It is a bit better when they can complete a Word document and submit via email with no signature. Either way, when you request past performance, you are generating lots of tedious work for lots of government staff.
- New entrant bias: The government contracting ecosystem needs fresh perspectives. The specifics of past performance requirements can push out some desirable firms, such as small businesses ready and eager to level-up, or leading firms who are new to the government space. In other words, it favors the entrenched “beltway bandits.”
The worst part about the recent procurement in question was that it was a task under an IDIQ. The bidders had already submitted past performance to be qualified for the IDIQ, and had to submit potentially the same set of past performance references again at the task level, partially defeating the purpose of the IDIQ structure.
When is Past Performance a good thing?
[Update: I added this section to address cases to reward earned advantage.] There are some cases where past performance can be quite valuable. Note also that in both of the below cases, Corporate Experience can serve a similar function.
The most obvious case is when there is an existing incumbent(s) doing great work. There is a well-deserved incumbent advantage in that case, and that kind of direct proven performance should be rewarded with continued work (assuming their price and other important parts of the proposal are within reason). In this case, a Past Performance evaluation factor can be a formal way to include that advantage in the scoring.
Similarly, if a firm has a great reference for very similar work elsewhere, and they happen to be delivering the same team, then that weighs heavily. This is a pretty rare case, as it is not common for a team to happen to free up just when another contract is set to start.
What’s the alternative?
So how do you get at the value of understanding the experience and performance of firms without using past performance in the evaluation?
- Corporate Experience: I’ve seen a number of recent procurements, from small to really big, that have no Past Performance in the evaluation. In these cases, they had some form of corporate experience highlighted. This gives you some sense of delivery capability or risk reduction.
- CPARS: The Contractor Performance Assessment Reporting System is a central database for contractor performance. Contracting Officers can fill out the form once and have it available to all other COs. It doesn’t give a great narrative insight into the nature of the work, but can certainly be a guard against the ‘cherry picking’ issue above. Note that you might want to allow firms to provide explanations for bad reviews. I’d view it more as a guard against bad actors than a identifier of the best qualified firms. [Update: I’ve gotten some feedback from COs that CPARS is relatively worthless, and bad performance rarely shows as such.]
- Ask more directly: The government typically asks for experience/performance because they care about how well the vendor will do on the next project. Instead of assessing this in the abstract, you can reframe the issue. For example, in the technical proposal instructions, you can explicitly instruct bidders to explain where they have applied this approach before and how that experience lowers delivery risk.
- Evaluate team members: Generally, the experience of the actual people on the delivery team trumps the experience of the firm. You can evaluate this in Key Personnel or a Management/Staffing Plan.
- Focus on the future: In some cases, the past may not be relevant. You may want to evaluate on forward vision, not the rear-view mirror.
It all boils down to a simple formula: figure out what you really need from the vendor for successful delivery, then find the best indicators of that future success. Each project is different, but getting a bunch of past performance questionnaires seldom gets you closer.
