Frequently Asked Questions (FAQs) about Applied Policy Research

General Concepts of Applied Policy Research 

Qn > What is applied policy research, and how does it differ from other social research?

Ans> Applied policy research takes theoretical concepts and puts them into practice to address real-world policy situations. Its primary purpose is not to develop general theories, but to generate empirically and logically credible lessons that can guide action to resolve specific public problems. It differs from other social research by its direct focus on organizational decision-making within real-world environments and problems influenced by human intention. 

Qn> What are the key principles that guide effective applied policy research?

Ans> The core principles include: 

>>  Be Real: Understand the problem, user, and researcher realities of each project. This means considering factors defining the problem, individuals/organizations using or opposing the information, and the researcher's own tools and professional considerations. 

>> Be Creative: Organize complex, real-world problems to make them amenable to systematic, flexible, and credible analysis. Creativity is necessary to align data with the problem at hand and to avoid conventional practice traps. 

 >> Be Credible: Select and use policy research tools and designs that provide strong arguments for credible and useful information. 

>> Be Useful: Develop and deliver actionable information that can transform technical data into user wisdom for making and implementing policy decisions. 

Qn> What are "tame," "messy," and "wicked" problems in policy research?

These terms categorize the complexity of policy problems: 

>> Tame problems are well-structured, static, involve relatively few variables and connections, and typically have a single perspective framing them, for which research procedures are readily available. One optimal solution may be expected. 

>> Messy problems are ill-structured, with more variables, more connections, less linearity, more contextual effects, and greater dynamism. These problems often have multiple suboptimal solutions. 

>> Wicked problems (or wicked messes) are highly complex problems that also involve numerous "humans in the loop" with competing interests, motivations, and values. They are particularly challenging to conceptualize and analyze, requiring policy researchers to engage with the messiness and wickedness to find actionable components. The case studies in the provided sources are predominantly categorized as messy, wicked, or wicked messes, with nothing being entirely tame. 

Qn> Why is "context" so important in policy research?

>> Context is a fundamental reality of social interaction, as it is nested within complex environments. Policy researchers must always consider the context of the problem they are studying, incorporating its constraints and opportunities into the research design. Understanding context helps in framing questions, selecting tools, and ensuring the correspondence of findings to reality. 

Data and Methods in Policy Research

Qn> What is the distinction between "harder" and "softer" data?

>> Harder data is typically quantitative (numeric), precisely defined, and comparable across multiple observations, suitable for statistical analyses. 

>> Softer data is qualitative (words or images), allowing subjects to express their own ideas and perceptions, providing insights into the experience of reality. 

The degree of "hardness" is not inherently better or worse; the optimal mix depends on the study's needs. For example, interview data, which is soft, can be "hardened up" by coding it into categories for analysis, while still retaining narrative detail for richer understanding. 

Qn>What are units of observation and levels of analysis?

>> Units of observation are the entities from which data is directly gathered (e.g., individual students in a survey). 

>> Units of analysis are the entities about which analytic statements are made (e.g., schools, if the survey data is used to describe school quality). 

Policy research often involves hierarchical structures, where units of observation are nested within higher levels of analysis (e.g., students within schools, or schools within districts). 

Qn> How do logic models contribute to policy research?

>>> A logic model is a graphic representation of an interconnected system designed to achieve a policy or program goal. They act as "blueprints" for tool selection by visually mapping components (concepts, structures, activities) and their logical flow, influences, or chronological progression. They help bridge real-world understanding to a research-ready conceptualization of the study problem and purpose. 

Qn>

What are the common types of research problems addressed by policy researchers?

>> Policy researchers typically address four general categories of research problems: 

    1. Exploration: Used when understanding of a problem area is minimal, aiming to build understanding or design a relevant study. It often employs qualitative techniques like interviews, focus groups, or document reviews. 

     2. Description: Aims to answer "What's going on?" by collecting and analyzing data on the world as it is, with minimal bias or control. It is often combined with pattern matching to describe conceptually what needs to be observed and why. 

    3. Causation (Effectiveness): Seeks to assess whether a policy intervention achieves its intended effects. While randomized controlled trials (RCTs) are ideal, quasi-experimental designs are more common in policy research due to real-world constraints. Pattern matching techniques can also be used to provide evidence of effectiveness. 

    4. Choice: Involves deciding among policy alternatives, often in complex or wicked problem contexts. Cost-benefit analysis is a quintessential, highly technical form of choice research, using monetary value as a common comparison criterion. 

Qn> Why is using "mixed methods" crucial in applied policy research?

>> Mixed methods involve combining qualitative and quantitative data and analysis methods within the same study. This approach is crucial because policy problems are typically too complex for a single research method to adequately document or measure. Mixed methods enhance the validity and objectivity of findings by incorporating multiple perspectives, providing more comprehensive pictures of how policies work, and promoting value-conscious research that acknowledges stakeholder differences. The case studies confirm that virtually every project uses mixed methods to a substantial extent. 

Applying and Communicating Policy Research 

Qn> How can policy researchers ensure their findings are actionable and used?

>> To ensure usefulness, policy research should: 

        - Be forward-thinking by working backward: Involve practitioners from the start to align research content with their information needs. 

        - Focus on the story, not just the design: Convey key findings and their implications through clear, narrative styles and impactful visuals, going beyond mere data presentation. 

        - "Define the "policy envelope": Clearly outline which aspects of the problem are amenable to policy manipulation. 

        - Be transparent about limitations: Candidly assess the study's strengths and limitations to provide context for decision-makers and prevent overinterpretation. 

        - Identify next steps: Propose appropriate next steps for the intervention or future research. 

        - Empower the user: Provide concepts, information, and perspectives that help users make better intentional decisions rather than simply dictating actions. 

Qn> What is "implementation fidelity" in the context of Evidence-Based Practices (EBPs)?

>> Implementation fidelity refers to the degree to which a program's actual delivery in a real-world setting matches its original program model. In linear approaches to EBPs, fidelity is a central criterion, often implying exact replication of a "manualized" intervention that has been rigorously evaluated. However, an agile approach emphasizes that implementation environments may require adaptation of the model to maximize effectiveness, balancing fidelity with local "fit". 

Examples from Case Studies 

Qn> How was mixed methods applied in the National Cross-Site Evaluation of High Risk Youth Programs (HRY)?

The HRY evaluation utilized a hierarchical, multisite, quasi-experimental, multimethod design. It incorporated detailed measurements of real-world program context, intervention design, implementation, and demographic/outcome data from participants and comparison groups. This enabled creative and agile policy research, moving between different levels of analysis to identify and confirm findings. The study also demonstrated a mixed-methods measurement approach to cement the correspondence to reality. 

Qn > How did the "Criteria Alternative Matrix (CAM) Analysis" help in the "What to Do About Scrap Tires?" case?

>> Professor Wassmer's team used the CAM analysis as a rational method to increase logical clarity in deciding on state subsidies for waste tire processors. The CAM involved defining the problem, assembling evidence, listing alternatives, selecting evaluation criteria, projecting outcomes, and describing tradeoffs. Wassmer modified it to include Likert scale ratings and relative weights for a quantitative comparison, which facilitated the confrontation of tradeoffs between policy alternatives.

Qn> What were the key takeaways from the "Transit Tax Initiatives" research regarding policy learning?

>> This case demonstrated a policy-learning program through a series of revisited, expanded, and replicated studies over an 11-year period. The research aimed to identify factors consistently associated with successful tax initiative campaigns across diverse communities, providing information on general conditions and actionable strategies. Despite limitations in providing explicit recommendations, the findings, based on a two-pronged approach of quantitative analysis and qualitative case studies, offered self-evident implications for decision-makers. The robust generalizability of the findings was a key aspect of their utility. 

Qn> What was the "bottom-up" estimation approach in the "High-Speed Rail Workforce Development" study?

>> Faced with a lack of existing research on high-speed rail workforce needs in California, the research team adopted a "bottom-up" estimation approach. This method involved identifying specific components of the complex HSR project, estimating the personnel requirements for each component, and then aggregating these estimates to produce a credible overall project workforce estimate. This creative response to a challenging information gap provided specific job types and required education/training backgrounds. 

Qn> How did the "Climate Change Adaptation" study ensure consistency in its multi-community case study approach?

>> The research team developed a research protocol to systematically identify and characterize factors shaping adaptation actions across 17 diverse communities. This protocol guided each researcher to explore essential elements of their case consistently, using a common heuristic. It provided a structure for compiling, assessing, and synthesizing data, adding rigor and quality control to the research process and facilitating cross-case analysis.