Patterns for Crafting Technical Challenges
Federal government agencies are adopting technical challenges as a way to evaluate proposals for application development contracts. Vendors are able to “show not tell” what they do (see also: DHS FLASH Primer). Much like paper-based requests for proposal, tech challenges can be crafted well… or poorly.
Below, I lay out my advice on how to design an appropriate challenge. Note: this is my first cut — challenges to these ideas and additional insights are more than welcome via comment.
Why Challenges?
Traditionally, would-be contractors submit a written technical approach. They write about how they will deliver, and are evaluated on that approach. Elaborate diagrams show what will happen and how. However, the people who write the proposal and create the graphics are often not who shows up to deliver. Firms even hire outside consultants to write proposals and create graphics for them. The content of proposals may have little correlation to actual delivery.
A technical challenge attempts to address this by evaluating something closer to the actual work. Generally, this means writing code instead of text.
Formulating a Strategy
First, understand the bottom-line capability you need from vendors. Is there a specific business problem to solve? Is a certain technical roadblock worrying you? Do you just want to see if they have “it”? What, in talking-to-a-friend terms, are you looking for in a contractor?
The challenge is an opportunity to get information about contractors — to answer some questions. What are those questions? If you cannot articulate those questions, do not proceed!
Challenge Design Options
Now you know what you need, let’s review some design tools at your disposal. The main design levers for a technical challenge are:
- Scope: How wide or narrow will the direction to teams be? Will teams implement a specific requirement, choose from a large backlog, or start with a high-level business problem? A very narrow scope for a challenge makes teams more comparable, but being overly prescriptive limits opportunities for innovation.
- Production-readiness: What aspects of the code should be “production ready”? Do you expect it to be secure, deployable, tested, etc? If you are looking for design and innovation, you may be asking for an interactive prototype and not production-ready code. Either way, be clear and ask for only for that which adds value to your evaluation.
- Scale: How much should teams get done? Are the building a small feature or half the system? Make sure you get the information you need to evaluate, but that the challenge is also worth the time of the firms.
- Duration: How long will teams have to complete the challenge? 2 hours? 2 weeks? What burden are you putting on teams? Also allow time for Q&A, etc. Many due dates are pushed back so that the government can answer questions.
- Observed: Do government evaluators watch the teams work, or simply review the work product? If it is in-person, are teams allowed to do prep, such as setting up technical “plumbing”? Will the team present their solution and approach in a 30–60 minute presentation?
- Technology: Do you specify a tech stack? You can specify agency technology constraints (e.g. must run on cloud.gov) or leave it open. Proposing team choices can act as a litmus test of sorts (an open source javascript stack may be fashionable and ‘modern’, while an Oracle-Java stack may be telling in other ways).
- Team Size: Do you limit the team size? Do you want to compare the effort put in across different teams? You may want to encourage vendors to use a representative team size to make the challenge closer to real performance.
- Representative Team: Do the challenge participants need to be the people who will deliver for the next several months? Do they need to be employees of the contractor or subcontractors? Seeing the real team may be helpful for a specific project, but not relevant for an IDIQ. (See also: Key Personnel in Tech Challenges & Procurements.)
- Down-select: Will you let all proposing firms participate in the challenge, or winnow it down to a few? Do you trust the down-select criteria to filter appropriately, or will it weed out the type of firms you actually want?
- Compensation: Will you pay for the challenge work? This is rare, but at least one procurement has paid firms a nominal amount for a small challenge. Small vendor pools, such as within an existing IDIQ, could make this more realistic — i.e. have 4 firms all perform a sprint or two worth of work in a paid task.
The Patterns
With so many options, what do you do? Let’s start with your needs simplified to two dimensions: certainty and scope.
Certainty refers to how known your solution is — or how unknown the unknowns are. (See also: Greg Godbout articulates the cone of uncertainty for app dev projects.) Highly uncertain projects do not have a well-defined problem, much less a solution. Highly certain projects have known problems and a known solution. Scope refers to how broad the project or set of projects may be — from a single project to a yet-undefined set of future programs.
Simple Execution
You have a set of battle-hardened technologies and processes. You need firms who can run with that playbook. Example: a maintenance contract for a well-functioning system and process.
Challenge Design: Vendors are given the tech stack (and maybe the environment), the application code, and a short backlog of changes. Have them work through the backlog in an observed 4-hour challenge. Grade the quality of the technical work product and how well they follow the process. The code should be production-ready, without cutting corners — even if the delivered scope is trivial. I’ve even heard of a challenge along these lines, where firms released code into production (or at least ran it through the pipeline to production)!
Technical Bake-off
You know what you need built, but have a very specific technical concern that is worrying you. You want a viable solution to a thorny problem. Example: you have to process a high volume of transactions each day involving high-resolution images, and need to verify that an architecture can support that volume.
Challenge Design: Vendors provide a technical proof of concept to directly address the central challenge. Limit the scope to that and provide explicit instructions that you will only evaluate the one technical piece. Do be open to innovation as appropriate (one vendor throws hardware at a performance issue, the other uses some compression strategies). Do not give points for visual design if that’s not where the risk is. Allow vendors to work offsite then present/demo the approach. Provide 2+ weeks to give scheduling flexibility for challenge staff. An accurate pre-RFP announcement of the challenge timeframe is also helpful.
Design Vision
You need a specific product delivered, and need a team to build it. For example, you are going to build a data portal to allow citizens to browse data produced by the agency. You care about the product concept and the ability of the team to deliver.
Challenge Design: You need a great product concept, and that will be the center of your challenge. Allow teams to choose the technology, perhaps with a few constraints. Give them 1–2 weeks to develop a functioning prototype to illustrate the product vision. Require that a couple of key roles (perhaps a Product Manager and/or Technical Lead) be committed resources.
You might observe the team, particularly if you want your Product Owner to participate and see with which team they can best collaborate. If the project is high-value, compensate down-selected teams to produce a prototype. You could first down-select on a small (5-page) product vision concept paper. Make sure you also understand (via paper or presentation) how they will ensure technical quality and incorporate real user feedback.
Long-term Collaborator
You need a set of vendors for an agency-wide IDIQ to provide application development support on a broad range of systems. A successful firm may end up with multiple teams. You need to understand the general capability of the firm.
Challenge Design: This contract supports multiple systems, so the scope can be anything. You need to understand the ability of the firm to discover your needs and collaborate with you. The challenge scope is a representative application. Don’t choose anything that favors incumbents too greatly (i.e. an existing agency app), particularly if much of the work would be new development. You could use a fictional app (twitter for inside the agency).
Observe the challenge in a 4–6 hour window. Grade teams on how well they collaborate and run an effective process. If the team can’t at least pretend to collaborate for 4 hours (among themselves and/or interacting with the client), they probably can’t do so in the real project. Limit the team size to 5–10 based on what you think is most representative of real work. Do not require participants to be key personnel, since this is an IDIQ/equivalent.
Blue-sky Innovation
Your agency can improve your mission with better applications. You want a firm to help shepherd creative new digital capabilities to better execute the agency mission. They will help with multiple projects, and even help discover and define new projects.
Challenge Design: Open up the scope to solving a very high-level problem (e.g. Improve TSA’s image with travelers, or improve CFPB’s ability to elicit consumer complaints). Teams will have entirely different approaches. Set clear evaluation criteria such as mission impact or number of stakeholders benefited. Observe the teams as they collaborate with real Product Owners and ideally real users. Relax constraints on technology and completeness. The important factor is to understand how they run an effective process to identify new innovations. (This in contrast to the ‘Design Vision’ challenge, where we care more about one vision for one problem.)
Feedback
Critical comments are welcome. This is a first draft of some ideas, so there are bound to be some logic holes, scenarios left out, etc.
