World Bank, 1818 H St NW, Washington, DC. Email email@example.com
The recent evolution of agricultural research systems in developing countries is described, noting a change in focus from national public sector research organizations to one that emphasizes a diversified public-private system, in which clients, especially farmers, are key partners in financing, planning, implementing and monitoring research. Consistent with these changes, research planning, monitoring and evaluation has evolved from centrally-driven top-down approaches to give greater emphasis to decentralized and participatory approaches, in which client demands and satisfaction are key inputs. Although much progress has been made in reforms to enhance accountability and impacts of research, res systems still face major challenges in ensuring that demand-driven approaches provide coherent research programs consistent with national policy objectives, and in ensuring that they reflect the interests of the poor and are not captured by elites.
Investment in agricultural research has been one of the major drivers of growth and poverty reduction in developing countries. However, many research systems in developing countries face a crisis of funding and sustainability. New approaches to organizing and managing research systems provide increased participation by farmers in financing, planning and monitoring of agricultural research.
Agricultural research systems, developing countries, research planning, priority setting, monitoring and evaluation, performance indicators
Investment in agricultural research systems has accounted for a substantial share of the unprecedented rate of agricultural productivity growth in developing countries over the past three decades. In the future, in an era of ever growing scarcity of land and water, ‘pro-poor’ agricultural growth will depend even more on application of new knowledge provided through national agricultural research and innovation systems.
Although the share of private investment in agricultural R&D is steadily rising, public research organizations still carry out over 90% of the R&D in developing countries (Pardey and Beintema, 2001). In today’s world of scarce public funding and greater accountability for these funds, governments, donors and research managers are increasingly demanding efficiency in the allocation of these resources, and assessment of the impacts of their investments. Although studies consistently show high returns to investments in agricultural research in developing countries (Alston et al., 2000), by the late 1990s, a slowdown in public spending on research had become a worldwide phenomenon (Pardey and Beintema, 2001). The most severe effects were felt in Latin America, Sub-Saharan Africa, and the countries of the former Soviet Union. This slowdown and continuing weaknesses of research systems to respond to a rapidly changing institutional, scientific and market environment have resulted in a renewed focus on ways to enhance accountability and impacts of agricultural research. Accordingly, approaches to planning, monitoring and evaluation (PME) of research have changed considerably in the past few years, in response to rapid evolution of the systems themselves. This paper traces that evolution at the system level, and discusses options for enhancing impacts and accountability of public research organizations.
The evolution of national agricultural R&D systems over the past four decades can be divided into at least three stages. A “bricks and mortar” period up to the early 1980s emphasized development of national public agricultural research organizations (NAROs) through consolidation of existing human and physical R&D resources into (usually) one public organization and further investments to expand and strengthen the breadth and depth of research in the NARO. From the late 1980s, investment began to slow, and emphasis shifted to improving efficiency of using existing resources through better PME, improved financial management, greater accountability, and increased relevance of programs to clients. In the mid- to late-1990s the continuing instability and inefficiency in many public research organizations led to a third phase, emphasizing greater private sector and client participation within broader agricultural innovation systems. Seven major challenges for national agricultural research systems provide the current context for developing improved approaches to ensuring greater accountability and impacts in agricultural research.
In the 1990s, emphasis has shifted from almost exclusive attention on NAROs to a broader focus on research systems, defined to include the NARO, universities and the private sector (both for profit and non-profit). Many countries are also promoting greater integration among research, extension and education organizations—the so-called agricultural knowledge and information system (AKIS). Others are using the concept of national agricultural innovation systems (NAIS) that expand this concept even further by recognizing wider sources of innovation (including farmers and foreign suppliers), and a non-linear pattern of interaction and feedback between research, development, and uptake of technology (Hall et al., 2003). The aim has been to (i) encourage participation of additional research funders and suppliers, (ii) bring more resources into the research system, such as well-trained scientists at universities, (iii) exploit complementarities among various research providers at the national and international levels, (iv) facilitate technology “spillins” through private agribusiness and international and regional networks, and (v) link research providers and users through a variety of uptake pathways.
With economic liberalization and globalization, R&D systems are confronting new priorities, especially (i) competitiveness of agriculture in local and international markets, through technologies that reduce production costs, improve product quality and food safety, and promote higher value and value added products, (ii) conservation of natural resources and the environment through sustainable land and water management, reduced agricultural pollution; and provision of environmental services through carbon farming and conservation of biodiversity, (iii) knowledge-intensive agriculture to use existing inputs more efficiently and sustainably, and (iv) poverty reduction by focusing on commodities, regions and technologies to maximize benefits to poor producers and consumers.
Investment in R&D by both the local private sector and multinationals is growing in most countries. As a result, the public sector is being challenged to more clearly define the public-good component of research—that is, basic and strategic research (long-term research with uncertain payoffs and high spillovers), research on problems of small-farm agriculture (high transactions costs for farmers to organize their own research), and research on natural resource management (positive environmental externalities). The challenge for the public sector is to facilitate private sector entry, promote competitive technology markets, provide strategic research backing, and ensure that its own activities do not undermine or ‘crowd out’ private markets. With the private sector increasingly serving the commercial farming sector, public funding is being challenged to focus more sharply on the needs of poor producers, especially in marginal areas.
Many public research organizations (i.e., NAROs) have suffered a financial crisis, resulting in erosion of operating budgets and salaries. The funding crisis in turn reflects the fact that many public research organizations suffer from weak leadership, bureaucratic and political interference, and low morale. The highly centralized organizations that resulted from the creation of the NAROs are now being questioned, especially on the bases of their ‘supply-led’ research orientation, and poor links to clients. These institutional and funding weaknesses have reinforced each other, leading to a loss of the best scientists and a downward spiral in many public organizations.
Responding to these difficulties, NAROs are attempting to streamline research priorities, reform management and incentive systems, and involve a broader range of actors, including farmers, in the research process. In many cases, this requires reforms to create flexible and efficient “autonomous” research organizations that are run along private sector lines, with independent governing boards representing key stakeholders. Many of these previously highly centralized NAROs are also looking at ways to sharply decentralize many of their research activities to regional or local governments, to facilitate better linkages with clients. However, “reform from within” has been slow, especially in the absence of a supporting policy environment with strong “champions” for agricultural science and technology.
To be effective and sustainable, research organizations are being asked to become more responsive and accountable to clients. Past investments in science and technology have mostly focused on supply of research products and have undervalued the pivotal role of farmers as purchasers, providers and co-financiers of research. The challenge is to effectively involve clients of the research system in order to generate a more demand-driven research agenda. The institutional models for achieving this include full or partial funding of research by farmers and other clients, involvement of farmers and farmers’ organizations in governance of research organizations, and various types of contractual relationships with clients in executing research. Decentralization of research organizations to put them closer to their clients, and empowerment of those clients ultimately lead to quite radical shifts in the flows of funding, with funding increasingly flowing to clients, who then contract needed research services. For example, producer organizations (POs) may receive matching grants to contract research organizations to carry out research on problems that the producers define. These institutional innovations also imply a shift in disciplinary balance away from research organizations dominated by technical scientists in the supply led models, toward more multidisciplinary approaches, involving especially social scientists.
In a pluralistic system, an additional challenge is to establish an efficient and effective division of labor for science and technology though partnerships to integrate various players into an overall system, exploit institutional comparative advantage and reduce costs. In a world of global science, links with international agricultural research—both in international research centers and advanced research centers in both the North and South—are essential as a source of new knowledge and innovations, especially for accessing new tools and products of biotechnology.
Likewise for technology uptake, linear systems of passing research results to extension agents who then transfer them to farmers are now widely regarded as obsolete. Agricultural extension systems are becoming much more pluralistic, with wide involvement of the private sector, including non-governmental organizations. Farmers themselves are also becoming better organized to contract and provide advisory and information services. R&D organizations are therefore exploring a variety of potential uptake pathways for their products. Delineation of likely pathways early in the research process is often critical to ensure that appropriate partnerships are established, that client perspectives are fully reflected in technology design, and that useful research products do not ‘sit on the shelf”. Both upstream and downstream partnerships present new challenges in term of capacity and skills for public-private collaboration, and the forging of regional and international alliances.
Developing countries invest on average 0.5% of agricultural value added in R&D compared with some 2.6% invested by industrialized countries. A major challenges for nearly all developing countries is how to increase public investment in agricultural research in order to promote a sustainable and competitive agricultural sector. A related challenge is how to downsize research organizations to fit the resources available, using additional funds for operating costs, and to pay scientists competitive salaries. A variety of new funding mechanisms, such as commercialization of research products and services, levies paid by farmers, and environmental funds, are being tapped to supplement public funding. Research funding and the setting of broad priorities is being performed by specialized funding bodies such as research councils that increasingly allocate funds through competitive and contractual mechanisms to encourage wider participation, competition, and accountability.
All of the above challenges reflect a much more complex institutional environment for research organizations, and even more so when they are seen as part of wider national innovation systems. Part of this complexity is seen in Figure 1, which shows the greatly increased number of actors and linkages in national systems, compared with early concepts of NAROs. These changes present new challenges for PME which has evolved considerably from approaches used in research systems built around NAROs.
Figure 1: Public Agricultural Research Funding
Priority setting and planning1 is carried out explicitly or implicitly in all research programs through the allocation of research resources to different commodities, regions, disciplines, problems, and types of technology. Priority setting occurs at various levels of decision-making—most commonly at the national, program, subprogram, and project levels (Table 1). Macro- or national-level resource allocations across major research programs and institutes provide the greatest strategic leverage of research resources through priority setting. In NAROs, diverse methods have been applied, from informal methods based on previous allocations (i.e., precedence), informal discussions and consensus among research managers taking account of sectoral strategies and priorities, and formal quantitative methods using crop loss studies, scoring models, congruence or economic surplus models, and geographical information systems (e.g., Hassan, 1998).
Table 1. Comparison of supply-oriented and demand-oriented approaches to priority setting
Major resource allocation decision
By commodity, factor or region
Economic surplus approaches and scoring methods
Limited. Subject to political processes and commodity/regional membership of governing board
By problem area for strategic research
Yield loss studies
Limited but may aggregate results of local diagnostic studies across regions
By region and technology type (e.g., variety, pests)
Yield loss studies, economic surplus and scoring models
Participatory planning across the market chain
Selection of technology characteristics suited to specific socio-economic and agroclimatic conditions
Scientists subjective judgment
Participatory rural appraisal, local farmer research committees
Consistent with the thinking at the time, and the emphasis on a public-sector monopoly NARO, PME was generally ‘supply driven’, centralized and carried out within a research organization. These central planning processes initially reflected informal processes based on national objectives, usually giving priority to improving self sufficiency in basic foods, as well as technical scientists perspectives on maximizing yields. Over time, considerable efforts were made to formalize planning and priority setting using economic criteria, especially the economic surplus approach (similar to benefit-cost analysis), but also various scoring models that were partially based on economic models. These methods, described in the landmark book, Science under Scarcity (Alston et al., 1995), were extensively incorporated into ‘master planning’ exercises for NAROs in the 1980s and 1990s, although few have institutionalized them as an ongoing process.
Most of these planning approaches have applied a variant of the economic surplus approach to allocate resources across commodities at the national level. Resource use efficiency was the main decision-making criterion, and commodities were ranked according to the net present value of the stream of benefits (net of research costs) per unit of investment in research on the commodity (Alston et al., 1995). However, these exercises were generally implemented by external teams of economists, with little ownership by senior decision makers and scientists. The general consensus is that they led to little effective reallocation of research resources in part due to lack of effective disciplinary interaction between technical and social scientists, entrenched bureaucratic interests, and difficulties of reallocating resources because of the specialized nature of scientific human resources and physical research infrastructure.
Monitoring and evaluation in NAROs was also centralized, usually focusing on monitoring inputs and expenditures, and initially with little emphasis on outputs and impacts. Some research organizations implemented external peer review of the technical quality of research, but these usually lacked a rigorous evaluation of the efficiency of the research in terms of benefits and costs, and effectiveness in terms of relevance of the research to clients. In the 1990s with the slowdown in investment in R&D, most NAROs initiated ex post evaluation studies to assess economic impacts, generally using economic surplus models based on benefit-cost analysis. Fairly standard and accepted methods are available, and are being increasingly applied for productivity-enhancing innovations such as improved cultivars, although simplistic assumptions about lags, costs, and supply shifts, together with failure to account for ‘research spillins’, have biased estimated RORs, usually upward (e.g., Maredia and Byerlee, 2000).
The emphasis on evaluating individual technologies in an ad hoc manner, rather than research programs on a regular basis has undoubtedly favored the selection of “winners” and hidden pockets of unproductive research. Impact studies have also been narrowly based on selected technologies, especially crop improvement, neglecting the growing allocation of resources to research on natural resources management, and the increasing focus on equity objectives. They were also widely criticized for a ‘black box’ approach that failed to rigorously attribute causality between research products and final benefits, especially for more complex technologies that could only be adopted in an appropriate enabling environment (Horton and Mackay, 2003; Ekboir, 2003).
Finally, impact studies have been largely carried out by NAROs to convince funding agencies of their efficacy, and not to provide feedback to management to improve the performance of the research organizations themselves. And while these studies did show generally high impacts, they had apparently little influence on increasing funding of R&D, in part because they usually evaluated research that was carried out a decade or more before impacts were assessed (Pingali and Morris, 2003).
Recently, emphasis in priority setting in research organizations has moved toward approaches that reflect the new emphasis on making research systems more demand-driven. In demand-oriented approaches, priorities are set based on perspectives of major stakeholders from outside the research system, especially farmers and other users. Demand-driven approaches have been most successfully applied at lower levels of decision making, especially in project selection, with major emphasis on choice of technology types and characteristics within a region or commodity. The move toward farming systems research carried out by multidisciplinary teams of technical and social scientists in the 1970s and 1980s initiated this orientation toward incorporating a client perspective in setting priorities for local-level adaptive research. These early approaches evolved from consultation with clients, toward more participatory approaches in the 1980s and 1990s that involve clients directly in setting priorities, and evaluating and selecting technologies (e.g., participatory breeding approaches) (Biggs and Farrington, 1991). Farmers now commonly participate in priority setting workshops (Janssen and Kissi, 1997; Mills, 1998) and increasingly are being empowered to make decisions about what research gets done and for whom. Farmer groups are conducting some adaptive research, making all major decisions and contracting technical input as needed (e.g., local farmer research committees—Ashby et al., 2001). Where strong POs exist, these organizations may play an important role in setting the research agenda, and even financing and executing much of the applied research (e.g., in Senegal (Rondot, Collion and Sarr, 2004), and commodity associations in several countries (Byerlee and Echeverria, 2002)). Farmers and farmer organizations are slowly gaining representation on governing boards or advisory councils of national and regional research institutes, where they can influence institute-level priorities. Consistent with a focus on product quality and markets, there are good recent examples (e.g., Colombia and Mexico) of organizing research along market chains from producers to consumers. This involves not only participation by POs, but also agribusiness, marketing and retailer associations.
Competitive funding of research, widely adopted in many countries in recent years, is also promoting demand-driven priority setting. Competitive funds increasingly require that farmers and other clients initiate research proposals and that funds (often in the form of matching grants) be provided directly to groups of clients who contract technical expertise to execute the research (e.g., as in Peru). Farmers also commonly participate in the governance of competitive funds in setting broad strategic priorities and, in some cases, also participate in the screening of research projects.
A major motivation for these changes has been the widespread perception that supply-driven approaches were not effective in reaching resource-poor farmers, especially in marginal areas. However, a continuing challenge for demand-driven approaches is also how to ensure the participation of the poor. It is widely recognized that participatory approaches are often biased toward rural elites (e.g., Biggs, 1989) unless conscious efforts are made to seek out and involve the poor, and scientists are provided appropriate incentives to work in field-oriented, problem-solving research with resource-poor farmers. Approaches that seek to empower farmers are even more likely to include these biases because leaders of farmer and community organizations are unlikely to be the rural poor. However, even imperfect participation and empowerment of the poor is likely to produce better results than conventional supply-led approaches on both efficiency and equity grounds, because it improves the probability of broad-based adoption.
A second major challenge is that priorities driven by thousands and even millions of small-scale farmers can lead to a fragmented and highly dispersed portfolio of research projects. Demand-driven competitive funds, for example, often result in a portfolio of projects that lacks coherence and that fails to exploit potential complementarities among projects. Similarly, such approaches may result in a research portfolio that does not respond to national policy objectives. Of course, demand- and supply-led approaches are not mutually exclusive. Workshop-type situations, in which clients and other stakeholders participate, can be used to define the main parameters for the economic surplus approach and arrive at final priorities (Mills, 1998). The challenge is to combine formal supply-led priority setting at the national level to allocate resources by broad themes and regions, with participatory approaches for determining program content at the local level. These approaches are likely to result in much better ownership of the resulting research priorities by major stakeholders and a greater chance that priorities will be translated into actual resource allocations.
Approaches to M&E have evolved in parallel with priority setting. In the new environment, research organizations are expected to be more accountable to key stakeholders, both to those who provide funds, as well as to their clients. In order for M&E to continuously incorporate broad stakeholder perspectives, recent approaches depend more on participatory processes and direct accountability to clients. For example, if funds are channeled through POs to contract needed research, then POs should have a major role in ensuring delivery of results. Matching grants with co-financing by POs provides incentives to POs to closely monitor sponsored research.
Evaluation of impacts, is also moving beyond narrow economic impacts to include other objectives, such as environmental and social objectives, and building capacity, human resources development, and institutional development (Goldsmith, 1993; Horton and Mackay, 1998). The methods for these impact studies are still evolving, but include client surveys and case studies (Horton and Mackay, 2003).
At the same time, in order for research organizations to adapt and change to a rapidly changing institutional, scientific and market environment, M&E processes must provide more immediate feedback to decision makers within the research organization to adjust program strategies and priorities, and research and uptake processes. This requires not only assessing the changing environment and continuing questioning of the relevance and utilization of research results, but also a better understanding of technology uptake pathways and a strong learning culture within an organization.
New approaches have evolved to allow research organizations to undertake self-assessments to guide adaptation and change based on outputs. These approaches define clear corporate objectives and organizational performance measures (Table 2) to assess intermediate results in order to provide feedback to decision making. Smith with Sutherland (2002) working in Ghana and Uganda, outline organizational performance assessment based on two categories of performance indicators; (i) external perspectives of clients and financial health, and (ii) internal perspectives related to business processes (e.g., technology release and adoption) and employee learning and growth. Peterson, Gijsbers and Wilks (2003) working in several Asian NAROs, developed a somewhat more internally-driven organizational performance assessment system based on defined outputs, productivity ratios in terms of outputs per unit of resource expenditure, and trends over time in these indicators (Table 2). EMBRAPA, the national agricultural research corporation of Brazil goes one step further in requiring all of its research institutes to assess their performance according to commonly-defined indicators with appropriate weights for each institute. Overall performance scores are then used as a basis for resource allocation among institutes. However, comparisons of performance indicators across research institutes with very different products and research production functions present serious methodological challenges. Also, using performance indicators to assign resources detracts from their original function of an internal process to guide research decisions and institutional change within an organization.
Table 2. Recent approaches to performance assessment of research organizations
Indicators related to:
ISNAR Organizational Performance Assessment System (Peterson et al. 2003)
Management processes scores
Assessing the organization’s context and responsiveness
NRI Performance Management Approach (Smith with Sutherland 2002)
Employee learning and growth
Internal business processes
Quality of research
Satisfaction of clients
Efficient use of resources
While there are merits in these new approaches to organizational performance assessment, the biggest difficulty in effecting institutional change in public research organizations is lack of incentives, especially low salaries and lack of rewards for good performance (and penalization for bad performance). These problems are outside of the control of research managers, in all but the most autonomous research organizations. Without tackling these more fundamental problems, many public research organizations will continue to decline in quality of human resources, and client relevance.
Recognizing the problem of incentives and lack of public funding, public research organizations almost everywhere have moved toward commercialization of both research products and non-research products and services, with a share of income generated being provided to individual scientists. The underlying rationale is that if a research organization begins to behave like a private-for-profit firm, it will be more efficient, and provide products relevant to the market. However, while sometimes providing much needed operating budgets and salary “topping up’, this strategy is likely to distort incentives toward providing private goods (e.g., hybrids), undermine private markets, and lead to conflicts with the broader objective of public organizations to maximize benefits to society at large (Fischer and Byerlee, 2002).
Although investment in agricultural R&D in developing countries has resulted in major successes, most developing countries have failed to develop effective systems to spur innovation in a rapidly modernizing agriculture, on a sustainable basis. This is reflected in a slowdown and some cases a decline in public funding, and a crisis in many public research organizations, which are no longer able to attract and retain the best scientists.
Against, this background, planning, monitoring and evaluation systems are in state of considerable change from highly centralized approaches within NAROs, to approaches that require greater accountability to clients and better evaluation of impacts. Participatory PME processes are gaining ground involving a wide cross section of stakeholderspolicy makers, research partners, and clients. Especially in determining content of adaptive research, recent experiences have shown the value of including farmers as full partners in financing, setting priorities, executing and evaluating research programs. While these approaches have led to greater involvement of economists and other social scientists in research decision making, most research organizations still lack the disciplinary balance and multidisciplinary interaction needed to promote more effective and bottom up PME.
Ultimately, the capacity of research organizations to learn and innovate in order to respond effectively to a rapidly changing environment, and the capacity of farmers and their organizations to plan and monitor research are keys to enhancing accountability and impacts of research. Within research organizations, a continuing challenge is how to integrate economic tools of supply-driven PME, with the new tools and mechanisms for demand-driven PME. The biggest challenge is to develop the capacity among key policy makers and stakeholders to promote change, learn from experience, and make adjustments in research organizations to meet changing demands.
Alston JM, Kang CC, Marra MC, Pardey PG and Wyatt TJ (2000). A Meta-Analysis of Rates of Return to Agricultural R&D: Ex Pede Herculem? IFPRI Research Report No 113, International Food Policy Research Institute, Washington D.C.
Alston JM, Norton GW and Pardey PG (1995). Science under Scarcity: Principles and Practice for Agricultural Research Evaluation and Priority Setting. Cornell University Press, Ithaca, New York.
Ashby JA, Braun AR, Garcia T, del Pilar Guerrrero IA, Hernandez LA, Quiros CA, Roa JI (2001). Investing in Farmers as Researchers: Experience with Local Agricultural Research Committees In Latin America. International Center for Tropical Agriculture (CIAT), Cali, Colombia.
Biggs S (1989). Resource-poor farmers participation in research: a synthesis of experiences from nine national agricultural research systems. OFCOR Comparative Study Paper 3, ISNAR, The Hague.
Biggs S and Farrington J (1991). Agricultural Research and the Poor: A Review of Social Science Analysis, International Development Research Center, Ottawa.
Byerlee D and R Echeverrķa (eds) (2002). Agricultural Research Policy in an Era of Privatization, CABI Publishing, Wallingford, UK.
Ekboir J (2003). Why impact assessment should not be used in impact evaluation and what the alternatives are. Agricultural Systems 78, 166-184.
Fischer K and Byerlee D (2002). Managing intellectual property and income generation in public research organizations. In Byerlee, D and R Echeverrķa (eds) Agricultural Research Policy in an Era of Privatization, CABI Publishing, Wallingford, UK. Pages 227-244.
Goldsmith A (1993). Institutional development in national agricultural research: Issues for impact assessment. Public Administration and Development 13, 195-204.
Hall A, Sulaiman, VR, Clark, C, and Yoganand, B (2003) From measuring impact to learning institutional lessons; an innovation systems perspective on improving the management of international agricultural research. Agricultural Systems 78, 213-242.
Hassan R (ed.) (1998) Maize Technology Development and Transfer: A GIS Application for Research Planning in Kenya. CAB International in association with CIMMYT and the Kenya Agricultural Research Institute (KARI), Wallingford, U.K.
Horton D and Mackay R (2003), Using evaluation to enhance institutional learning and change: recent experiences with agricultural research and development. Agricultural Systems 78, 127-142.
Horton D. and Mackay R (1998). Assessing the institutional impact of development cooperation: A case from agricultural R&D. ISNAR Discussion Paper No. 98-2, ISNAR, The Hague..
Janssen W and Kissi A (1997) Planning and Priority Setting for Regional Research: A Practical Approach to Combine Natural Resource Management and Productivity Concerns. Research Management Guidelines No 4, ISNAR, The Hague.
Maredia MK and Byerlee D (2000). “Efficiency of research investments in the presence of international spillovers: Wheat research in developing countries.” Agricultural Economics, 22: 1-16.
Mills B (ed.) (1998) Agricultural Research Priority Setting: Information Investments for the Improved Use of Research Resources, ISNAR and the Kenyan Agricultural Research Institute, The Hague and Nairobi.
Pardey PG and Beintema, NM (2001). Slow Magic: Agricultural Research a Century after Mendel. International Food Policy Research Institute, Washington.
Peterson W, Gijsbers G, and Wilks M (2003). An Organizational Performance Assessment System for Agricultural Research Organizations: Concepts, Methods, and Procedures, ISNAR Research Management Guidelines No. 7, International Service for National Agricultural Research, The Hague.
Pingali, P and Morris M (2003). Assessing the impact of agricultural research: An introduction. Quarterly Journal of International Agriculture 42: 121-125.
Rondot P, MH Collion, and L Sarr, 2004. Farmer-Driven Agricultural Services’ System: Lessons from Senegal, Draft paper, Agricultural and Rural Development Department, World Bank, Washington.
Smith, DR with Sutherland A (2002). Institutionalizing Impact Orientation: Building a Performance Management Approach that Enhances the Impact Orientation of Research Organizations, Natural Resources Institute, Chatham, UK.
1 Priority setting and planning are used interchangeably in this paper, although it is recognized that priority setting is only one element of research planning, that also involves setting broader strategic objectives as well as specific resource allocations to research programs, and decisions on investments in physical and human resources.