The truth about evaluation processes

I was involved in many evaluation processes when I was a Consultant Project Manager. Over the last 10 years I’ve been able to contextualise why I made the decisions I made and behaved the way I did. In this blog, I will share (candidly) some of the things I did (and witnessed) as an evaluator and provide some context.

When we understand how people evaluate, and why they react and behave the way they do, we can allow for it in our capture and sales processes.

Not all evaluation processes are the same

There are many variables:

• Formality (is there a probity auditor, for example)
• Personnel (are the evaluators experienced, or do they have subject matter expertise)
• Incumbency (is the current provider amongst the tenderers)
• Complexity of buying group (how many people will sign off the recommendation)

Depending on the structure and circumstances of the process, the evaluators are trying to solve for different challenges of their own. So, understanding their situation can help you know what your proposal must look like to be successful.

It’s an opinion, not a scientific conclusion

Evaluation processes are subjective. Full stop. End of sentence.

No matter how much procurement may protest that they have robust systems, the fact remains that evaluating proposals is a human process. And humans are emotional, instinctive, and biased.

Shipley has run a long-term evaluation experiment, with thousands of data points, where participants compare two proposals. One proposal has average to above average application of proposal best practice. The other proposal (while offering a slightly better technical solution) is a poor-quality proposal. Consistently, the same outcome occurs. The better-quality proposal wins the mock evaluation. We also see enormous variation in scores… some harsh, some generous.

Most bias is removed from this exercise, as the vendors and purchaser are fictitious, and the subject matter is foreign to most participants. So, this helps validate proposal best practice.

The wild variation in scores is a stark reminder of the subjectivity.

Here are some of the behaviours I observed in myself and others (and some explanation behind those behaviours) over my time as an evaluator.

I was lazy

I always looked for a short cut. Most (if not all) evaluators do. I looked for the quickest way to reach a justifiable score.

“… Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.”- Herbert A. Simon, Nobel Prize recipient

There is simply too much information in a tender evaluation process for one person to digest. So, the evaluator will focus on a few things, they will then craft a score to favour the tenderer who responds to these things best. Sometimes this is conscious, sometimes it’s subconscious, but it always happens.

The score outcome will depend entirely on the issues that matter to the individual. This is why a Win Strategy, based on sound customer knowledge, is so important.

My emotions got the better of me

Once, I rejected a tender because it was submitted in double sided A3. That’s not the official justification I gave, but it was the reason.

Why? Because it didn’t fit on my desk.

To evaluate it, I had to undock my laptop, find a meeting room, and relocate myself there so I had enough desk space to open and read the document. The tenderer had lost before I got to the meeting room.

If the evaluator doesn’t like you, they won’t score you well. Make your proposal easy to evaluate and the evaluator will thank you with high scores. To do that, you must be:

• Compliant
• Clear
• Concise
• Correct

It’s the questions we don’t ask

One of the biggest contracts I awarded was for a major construction project in Melbourne. It was a tough project. The winner had responded clearly to the problems we were facing, including the problems we didn’t mention in the RFT or the briefing sessions.

Their knowledge of a specific stakeholder management problem was a key difference. This lead them to win with less experience/expertise and a higher price than their competitors. They obtained this differential knowledge through relationship, trust, and ability to listen. 

There are many reasons, the customer will not include all the details about every problem they are trying to solve. If you can identify the problems that other tenderers are unaware of, you can offer value above and beyond a compliant solution.

Once you solve more problems than your competition, your price is assessed differently.

Which leads me to the subject of price.

Scoring price is more like magic than accounting

Let me run a scenario past you. The customer’s budget is $10m. They receive five compliant tenders for the following amounts:

• $9.9m
• $9.6m
• $9m
• $8.5m
• $8m

What score (out of 10) should be applied to these prices?

They are all within budget, so are they all 10/10?

Does the $8m price get 10/10, and the others get scored proportionate to that? If so, does the $9.9m offer get a 1/10, or a 9/10?

Scoring price is, in my experience, the most subjective of the whole evaluation landscape. In the scenario above, there is logic to say that you could mark the $8m price down because it’s too risky.

Ultimately, your price will be judged in the context of benefits your proposal can demonstrate.

If all the tenders demonstrate relatively equal benefits, then price will become a big factor. If the evaluators see value in an expensive offer, they will find a way to make the price criteria matter less.

Don’t discount to win. It’s not good business. Instead, work out where you can provide benefit that no one else can and highlight it. But most of all, make it easy for the evaluator to score you, and they will find a way to give you a good score.

The best way to do that is through Planners and Organisers. These help to build compliance, responsiveness, and customer focus into your proposals.

Evaluators are people just like you or me, so make their job a bit easier and it'll be reflected in your proposal scores.