Large organizations have numerous departments or teams that perform different functions, including Research and Development (R&D), Production, Purchasing, Marketing, Human Resources (HR), and Accounting and Finance. While many teams respond well to traditional management techniques, including setting milestones and reporting their progress, data science teams do not. Their purpose and function is more akin to intelligence agencies, such as the CIA, than to typical business units. Their success isn’t measured in milestones or productivity but in knowledge and insight. As a result, they need to be “managed” in a different way.

The Wrong Way to Manage a Data Science Team

In many organizations, teams are focused on setting and meeting goals and objectives. Managers spend most of their time planning, monitoring, and correcting to ensure compliance. They have quarterly budgets and monitor them closely. They look for cost or schedule variances. If they notice deviations from what’s expected, they track down and address the cause(s) or consult with the executive team for guidance.

This approach to productivity isn’t well suited for data science teams, because their work is primarily exploratory. The data science team asks compelling questions, gathers and analyzes data, develops theories (hypotheses), and conducts experiments to test its theories. Their “product” is an ever-expanding body of organizational knowledge and insights along with, perhaps, data-driven tools to automate and optimize certain tasks.

The Right Way to Manage a Data Science Team

Some organizations, such as pharmaceutical or high-tech companies, are accustomed to working scientifically. They are engaged in a constant cycle of insight — gathering and analyzing data, asking questions, formulating hypothesis, and testing those hypotheses through experimentation. For most companies, however, exploratory work is a foreign concept, and having a data science team engaged in exploration to create new knowledge just doesn’t seem natural.

In “rank-and-file” companies, getting a data science team up and running is especially challenging. You can expect to encounter institutional pressure to maintain separation between the business and the technology of data science. You can also expect a strong push to place a compliance manager (a project manager or director) in charge of the team. Either of these two approaches would significantly slow the pace of discovery.

In a previous post, “Building a Data Science Team,” I recommend creating a small team of three to five individuals, including a research lead, data analyst, and project manager. I also recommend adding people to the team on a temporary basis from different parts of the organization to benefit from different perspectives. This team should be given some level of autonomy, so it feels free to explore, but it should also work closely with other stakeholders to ensure that it serves the organization’s business intelligence needs.

I once worked for an organization that didn’t see the value of having a research lead on the team. They stacked the team with data analysts, who were expected to deliver monthly insights to the business manager who would then decide which insights to act on. The business manager had her own budget and wasn’t really interested in digging into the data. Her primary focus was to comply with her budget constraints. The data analysts had very little insight into the business, so they weren’t geared to ask compelling questions. As a result, the two teams functioned independently, never harnessing the power of the organization’s data.

Another company tried putting a project manager in charge of the data science team. His focus was on ensuring that the team met its objectives, and he developed different ways to measure the team’s output. At one point, he turned questions into tasks and then measured how well the team completed its tasks. This approach failed, because the questions led to more questions, which resulted in a growing list of tasks. The more the data science team did, the more it had to do, so deadlines kept slipping. His goal was to have the data science team complete as many tasks as possible, which doesn’t align with the value proposition of a data science team — to deliver valuable business intelligence.

Tips for Installing and Managing a Productive Data Science Team

Here are a few tips for making a traditional organization more agile and data-driven:

• Get executive level buy-in. Any major organizational change is more likely to succeed when top level leadership sponsors it. The organization’s executives must appreciate the value of making data-driven decisions at all levels of the organization, especially at the top.
• Work together on the data science team. Each data science team member should have specific responsibilities, but they should also work closely together to provide feedback and support. For example, if the data analyst is stuck, the project manager may have ideas for accessing additional data sets, or the research lead may ask questions that steer the data analyst in a different direction.
• Create dynamic feedback loops on the team. Everyone should work together to question, research, and learn. The team will always do better when everyone is participating in a lively discussion of discovery.
• Serve the stakeholders. The data science team’s mission should be to serve the stakeholders in the organization by providing them with business intelligence that informs their decisions. If the data science team does this well, it will find all the support and freedom it needs to mine the organization’s data for knowledge and insights.