Ben's Page

Executive Summary

We are all familiar with horrible software & workflow tools (e.g. Concur, Blackboard, Moodle, etc.). These tools frustrate users and make it difficult for teams to accomplish their goals, which begs the questions: what are the underlying causes of bad software and how can they be avoided?

In this post, I share an opinion on a few causes for terrible software & workflow tools and, more importantly, what business teams can do to avoid perpetuating the cycle of these tools. The main points in this post are highlighted below. A more in-depth perspective follows.

Underlying Causes for Terrible Tools

Disengaged leadership
Being overly focused on the cheapest, fastest ‘solution’
Emphasizing desired features rather than articulating challenges and desired outcomes

Techniques Businesses Can Use to Avoid Building Bad Tools

Assign an end-user to the tech-vendor team who is building the software/workflow tool
Work with the vendor to ensure they fully understand the business problem
Scope contracts to solve the smallest complete problem, and then scale the approach used to solve that problem

A Case Study on the Cost of Poor Software

In NYC, every K-12 student with special needs receives an Individualized Education Program (IEP). Several of the services required by IEPs are billable to Medicaid, so it is important for special education teachers to be able to easily and clearly document student progress along IEPs.

As a result, in 2008, the New York City Department of Education (NYC DOE) contracted Maximus, Inc. to replace NYC’s old paper-based system for tracking IEPs with a digital solution. The system was essentially meant to be a ‘digital document management system’ for the development, tracking, and documentation of student IEPs. Three years and 69 million dollars later, Maximus delivered the Special Education Student Information System (SESIS).

SESIS is a triumph of bad software. It has been a failure in almost every aspect imaginable. SESIS cost the city tens of millions of dollars in settlements, hundreds of millions of dollars in Medicaid reimbursements, malfunctioned more than 800,000 times a day, and made it difficult to track if students with disabilities were getting the services they need. After paying almost 50 million dollars to try to fix the system, the NYC DOE finally gave up and decided to scrap SESIS and start over.

The below articles shed more light on the troubling history of SESIS. To summarize the idea behind each, a few lines the articles are copied and pasted below each link. I’d like to note that having personally seen SESIS and talked with Special Education teachers, SESIS really is this terrible. None of the comments are exaggerations.

This $69M Special Ed Software Has Become NYC’s Own HealthCare.gov-like Disaster

“It robbed from both special educators’ instructional and personal time. Even today, the lousy user experience design of the software jumps out even without logging in”

“SESIS has been so cumbersome that, in May 2013, an arbitrator awarded teachers who worked with SESIS $38 million to compensate for unpaid overtime incurred by teachers wrestling with the platform outside of work hours.”

“Objectively, SESIS makes it difficult to impossible to know how well the school system services students in special education overall”

Say goodbye to SESIS

“SESIS was a source of constant frustration for our members. They toiled countless hours entering data — an investment of time and energy that did nothing to improve the education or well-being of the students they served” - UFT President Michael Mulgrew

“From the start, it was plagued with breakdowns — thousands reported in a single day — redundancies and slowness caused by heavy traffic that forced teachers to log data for thousands of hours outside school hours, including on weekends and holidays. These glitches were responsible for the loss of $373 million in Medicaid reimbursements to the city.”

“Rather than bringing providers together to discuss children’s needs, SESIS isolated them.”

Underlying Causes for Terrible Tools

1. Disengaged Leadership

A leader is responsible for anything his or her team accomplishes or fails to accomplish. No excuses. The building and implementation of bad workflow tools is therefore a reflection of leadership. I realize that is an aggressive opening statement, and certainly hope it doesn’t offend anyone reading this. It does, however, need to be said. Most tech failures stem from a lack of engaged leadership.

Note the emphasis on the word engaged. Contrary to popular belief, leaders to not need to be the ‘smartest people in the room.’ They do not have to be most tech savvy on the team, and they absolutely do not need to be a subject matter expert in any of the areas the tech-vendor will be working in. Leaders do, however, need to be engaged, because even the very best vendors will mess up if left to develop in isolation. Here are a few ways a leader can meaningfully engage the teams they’ve commissioned:

a. Articulate success clearly. This topic is so important that I dedicate a few paragraphs just to it in the coming section…

b. Ensure vendors have an adequate understanding of the business problem. I simply cannot overstate how important it is for the tech-vendor to have an understanding of the business problem being addressed. Without a shared understanding of the problem, there is a good change vendors will build excellent technical solutions to the wrong specific problems.

Towards that end, I would also advise clients to be suspicious of vendors who show up on day one with a pre-packaged ‘solution’ to the challenges a company faces. If a tech-vendor replies to a request for proposal (RFP) with a pitch that includes rhetoric about how they already have an 'offering' in place or are subject matter experts in your particular industry, proceed with caution. Such statements can be likened to doctors who claim to sell a ‘cure’ before having diagnosed their patients. Clients should instead be looking for vendors that are:

Awesome in their fields (programming, data engineering, data science, etc.)
Quick learners (who will be able to understand the business domain relatively quickly)
Strategic thinkers (who can help their client with the challenges in their field)

c. Communicate risks, concerns, and empower people to address them. Very few people go to work with the intention of doing a bad job. Quite the opposite, most people feel great when they know their efforts are impactful. This is why it is so important not just to give vendors checklists and to-do rosters, but to make sure they truly understand business concerns.

Keep in mind these vendors want to impress clients. They want clients to tell their friends and refer them with warm regards. But if vendors are simply handed a checklist of features, that’s exactly what they’ll deliver.

d. Articulate reporting requirements and cadences early in the work. This idea is straight-forward, but important nonetheless. Before tech-teams become too involved developing, business owners should clearly announce when they expect update reports/meetings, what info they want included in those reports/meetings, and in what format (live demo, PPT, etc.)

e. Ask for demos and educational sessions. Quite often, businesses will commission a tech-vendor to build software, data models, etc. based on their own limited understanding of what is possible. In doing so, businesses miss a lot of opportunities. As simple as it sounds, the best way to mitigate this risk is to ask for demos. Good vendors should be able to quickly demonstrate best in class tools, techniques, and procedures to their clients. In doing so, they often introduce their clients to new ideas and possibilities.

f. Set up time a routine time for un-structured, PowerPoint free, conversation. PowerPoint is a great tool for briefs and routine correspondence. But in the consulting world, PowerPoint is often a substitute for competence. Many consulting firms build extravagant PowerPoints briefs to mask the fact that they don’t know what they are talking about. They brief well, but there is little substance behind the glittery slides. Savvy clients carve out time to talk with their vendors in a completely personal manner: no PowerPoints, no spreadsheets, no fluff. Only meaningful, person-to-person conversation.

2. Being overly focused on the cheapest, fastest ‘solution’

To highlight this point, consider the U.S. Special Operations Command. (SOCOM). SOCOM is responsible for executing the most important missions to America’s national security. These missions are extremely difficult, dangerous, and are often realized in the aftermath of a unforeseen event or national emergency. Yet, SOCOM does not rush or scramble to develop solutions. They maintain a vision, a clear focus on success, and thoughtfully consider their approach before executing. In fact, SOCOM subscribes to five ‘truths,’ the fourth being ‘Competent Special Operations Forces cannot be created after emergencies occur.’

All too often, however, businesses try to create solutions after emergencies occur. They panic and rush to find the cheapest, fastest, ‘solution’ to their immediate problem. Naturally, they get what they pay for: patches to a near-term problems that fail to address the root causes of the issue. They get rushed jobs with sloppy, undocumented code that nobody understands. When that code breaks, the business panics and repeats the process. This cycle, over time, creates a convoluted, unclear infrastructure that causes more problems than it’s worth. To avoid this spiral, slow down.

Slow is smooth, and smooth is fast.

3. Emphasizing desired features rather than articulating the problem and desired outcome

Handing a tech-vendor a list of features to develop in lieu of articulating problem(s) and desired outcome(s) is a recipe for disaster.

Would any sane adult walk into a doctor’s office, not explain their symptoms, then tell the doctor to operate on them, and how? Medical professionals would clearly reject such a suggestion, because they don’t blindly accept a patients’ self-diagnosis on faith. Doctors, after all, are subject matter experts and are bound by oath to do what is best for their patients. They therefore carefully assess their patients problems and educate their patients on options before taking action.

Businesses, however, make these type requests of tech-vendors all the time. They release terse requests for proposals (RFPs) that give no context and demand a list of features they’d like developed and how they should be built. Unlike doctors, however, tech-consulting firms are not bound by ethical oaths to do right by their ‘patients.’ Sadly, most consulting firms lack the integrity to challenge such requests from clients. They gladly bid on such RFPs, even if they know what the client is asking for is bad for them.

Business are wise to outline the challenges they face and let tech-vendors respond with a ‘diagnosis’ and suggested course of action. Even if the proposal seems relatively straight-forward, smart businesses avoid the temptation to simply solicit responses on feature development. When given a problem statement, strong tech-vendors will often surprise with their creative, thoughtful perspectives.

Businesses Can Use to Avoiding Building Bad Tools

1. Assign an end-user to the vendor team who is building the tool, model, etc.

Napoleon Bonaparte reportedly kept a battle-hardened, front-line Corporal on his planning staff for sole purposes of receiving war plans. Before Napoleon or his staff would proceed with a plan, they would brief it to the Corporal. If the Corporal was able to understand the plan, and explain it clearly to other front-line soldiers, then Napoleon accepted the plan was simple enough to be executed. On the other hand, if the Corporal was confused by the plan, or had difficulty relaying it to others, then Napoleon would scrap the plan and seek to simplify it. This concept holds true in developing tools, models, etc.

Arguably the most important technique to avoiding the development of bad tools is to have an end-user on the team. The user does not need to have any experience in the development process. In fact, it is reasonable to argue a person without technical expertise is even more desirable for this role. If such a person is able to understand the tools being developed by the tech team, then any of their colleagues should be able to as well. At every step in development, the tech-vendor should be working with the end user to ensure the work makes sense from the perspective of the person who will be using it.

To highlight this point, consider this quote from Kesha Hill, a speech teacher from a public school in Fort Greene who has to use SESIS: “I’m afraid what comes next will be just as ineffective, unless the creators of the new system talk to those of us who work in it daily.”

2. Work with the vendor to ensure they fully understand the business problem

As mentioned previously, it is unreasonable to expect that a tech-vendor will be a subject matter expert in a particular (business/government) domain. They should be fast learners and strategic thinkers who are excellent in their particular technical fields.

But the better a tech-vendor understands the environment, challenges, and desired outcomes of their client, the more they can contribute. Savvy business, therefore, do not only ask for reports and presentations from their vendors. They also give reports and presentations to their vendors. These businesses help their vendors broaden their domain knowledge because they recognize time spent educating their vendors is an investment in their own business.

3. Scope contracts to solve the smallest complete problem, and then scale the approach used to solve that problem

I’ve observed this to be a particular problem when government agencies contract tech-vendors to build software, databases, or data models. I suspect this is a byproduct of the cumbersome contracting protocols commonly used by government agencies. Because government agencies use such rigid contracting protocols, they tend to try to jam everything into ‘super-contracts’ which take months or years to complete. By the time tech-vendors finish these contracts, the software/tools they were commissioned to build are either obsolete or too large to redevelop if they dont work. If this sounds too hypothetical, consider this horrifying SESIS example above.

I have reason to believe the contract to build SESIS was a full-scope contract. In other words, the New York City Department of Education (NYC DOE) contracted a tech-vendor to build the entire SESIS tool, apply it to all schools, for all students, and for all situations and special needs. Yikes. No wonder it took three years to complete.

A better approach would have been to contract a tech-vendor to create a solution for a very small subset of the broader problem. This solution should be designed to be tested and scaled rapidly through additional contracts. In the case of SESIS, the NYC DOE could have pursued an initial contract to solve for a very specific need in a single school. Once tested by special education teachers of that school, they could expand it for all schools with additional contracts. Then the approach used for that single special need could be repeated for additional student needs, and on and on until the final outcome was reached.