Review of NGO performance research published in academic journals between 1996 and 2008

Since the 1990s, the role of development non-governmental organisations (NGOs) in international development has increased, along with massive interest and concern over NGO performance from NGO practitioners, governments, citizens, donors, policymakers and academics.1,2,3,4 The growth of research on NGOs is relatively new,5 with development NGOs making significant efforts to demonstrate their performance. These efforts are as a result of stricter official aid requirements, as well as the NGOs’ increased desire to know what is being achieved for accountability, selfmotivation and improvement.6,7


Introduction
Since the 1990s, the role of development non-governmental organisations (NGOs) in international development has increased, along with massive interest and concern over NGO performance from NGO practitioners, governments, citizens, donors, policymakers and academics. 1,2,3,4The growth of research on NGOs is relatively new, 5 with development NGOs making significant efforts to demonstrate their performance.These efforts are as a result of stricter official aid requirements, as well as the NGOs' increased desire to know what is being achieved for accountability, selfmotivation and improvement. 6,7holars have called for further and more thorough research on NGOs, cautioning that the trends in NGO performance are based on ideological grounds in the absence of empirical verification. 8cholars further argue that inadequate research on NGO performance would cause this area of study to remain immature, make it difficult to get accurate and comprehensive data and thus adversely influence funding decisions. 4,9,10Intensification of research on development NGO performance can be attributed to Edwards and Hulme 10 who, in the mid-1990s, assembled a collection of studies on NGO performance.However, little is known regarding the distinctive characteristics of the existing body of NGO performance literature.
The diversity of NGO typology is a complex, contested and controversial matter.For the purposes of this paper, development NGOs refers to a diverse set of institutions (from grass roots or community level to national, regional or international level) that operate on a nonprofit basis, generally serve the public sector, and are engaged in long-term development work within the framework of international development cooperation. 9As the primary mission of development NGOs is focused on development goals desired by their targeted beneficiaries and their communities, the performance of NGOs should be assessed by their effectiveness as organisations to assist beneficiaries achieve mutually identified social goals.Traditionally, the evaluation results of NGO interventions have been utilised as a proxy for the overall performance of the organisation.There is now a call for NGO performance measurement, beyond project assessment, 6 to focus on the overall functioning of the organisation.
In a recent systematic review of journal articles published in online English-language academic journals between 1996 and 2008, factors influencing development NGO performance were Globally, literature on the performance of development non-governmental organisations (NGOs) has increased.However, little is known regarding the distinctive characteristics of academic articles on factors influencing NGO performance.In a recent systematic review of research, published in English-language academic journals between 1996 and 2008, factors influencing NGO performance were investigated.From the 31 journal articles that met the inclusion criteria, this study examined the salient characteristics of NGO performance research in terms of, (1) the number of publications, (2) publication outlets (journals and journal cluster), (3) author collaboration (sole or joint authors), (4) author affiliation, (5) study location, (6) study period, (7) study topics and ( 8) method and sources of information.Findings showed a steady increase in the number of articles, published in a wide array of journals with over half of the articles published in development studies journals.Of the 31 articles, 21 were sole authored.Data were mainly sought from NGO directors, programme staff and donors; comparatively fewer studies collected data from beneficiaries.Studies were mainly conducted in developing countries, whilst most authors were affiliated to institutions in developed countries.Of the 13 authors who conducted studies in Africa only 3 were affiliated to an institution in Africa.This study confirmed the continued need for increased research on factors influencing NGO performance; revealed the low seeking of beneficiaries' perspectives in NGO performance research despite the rhetoric of participatory development; and revealed the low number of published researchers in Africa and minimal collaborative efforts between 'Northern' and 'Southern' researchers in this field.
S Afr J Sci 2012; 108(11/12) http://www.sajs.co.za investigated. 11This paper takes the literature review further and focuses on the research characteristics of the 31 articles that met the inclusion criteria.Special attention was paid to, (1) the number of publications, (2) publication outlets (journals and journal cluster), (3) author collaboration (sole or joint authors), (4) author affiliation, (5) study location, (6) study period, (7) study topics and ( 8) method and sources of information.Findings, which will inform future research and policy directions, confirm: the continued need for increased research in NGO performance; the comparatively low reporting on beneficiaries' voices in NGO performance research; the low number of published researchers in Africa; and minimal collaborative efforts between the 'North' and 'South' in this field.

Methodology Search strategy
The search strategy aimed to identify, appraise and summarise papers published in English-language academic journals, which present findings on factors influencing the performance of development NGOs.The review followed the Cochrane Collaboration guidelines for undertaking extensive systematic literature reviews. 12The literature review was conducted in phases and involved a thorough search to identify relevant published studies, retrieve them and document the search process.Effort was made to ensure the process was comprehensive, precise, thorough and unbiased, to limit systematic error or bias in reviewing the available publications. 12,13,14great deal of time was spent becoming familiar with search parameters specific to the subject literature, and periodically consulting experienced librarians at the University of Cape Town.The initial attempts did not produce the final search strategy -the design progressed as knowledge was gained of the platforms, databases, keywords, indexing and structuring of texts.Internet searches were conducted using Google Scholar.Further searches were undertaken of those journals from which articles were accessed.Once relevant publications were sourced, their reference lists (and references of references) were examined and followed up for further relevant articles.Contact was made with experts in the field who suggested additional sources such as articles accepted by journals but not yet published, and names and email contacts of authors who had published work that may contribute to the review.

Inclusion and exclusion criteria
The inclusion criteria were: • Articles published between 1996 and 2008 in scholarly peer-reviewed journals.

Appraisal system
In the search and selection process, a standardised system was established to systematically accept articles in order to minimise the risk of error of judgement.The search for publications on NGOs found 14 469 citations.

Data extraction and synthesis
Hard copies of the articles that met the inclusion criteria were obtained and reviewed in detail.A tabulated framework for quantitative and qualitative data extraction was designed with which relevant information was populated.This information included administrative details such as author, year and journal of publication, location of author-affiliated institution, location of study, and period of data collection.When articles did not explicitly provide information on the period of data collection, the authors were contacted to obtain the information.
A grid was developed in MS Excel which was populated with the extracted data.This grid was exported to Statistical Package for the Social Sciences (SPSS), 15 a statistical software package for data analysis, where variables were defined, coded and screened for detailed and complete information.Data were 'cleaned' by checking each variable for consistent and accurate information.Frequency analysis was mainly utilised to generate descriptive statistics. 16,17In this process, new variables, such as number of authors, journal cluster, and author affiliation and study location (developing or developed country) were created from the initial administrative data set.

Academic articles on factors influencing NGO performance
The systematic literature search found 31 articles.The administrative details of these articles were analysed and are presented in this study.Discussions of the content of these articles are published elsewhere. 11er the years, there has been a steady increase in the number of academic articles analysing factors influencing NGO performance (Figure 1).The cumulative analysis shows that only 3 articles were published between 1996 and 2000, which increased to a total of 11 articles by 2004 and to 31 articles by 2008.Most studies (20) were published between 2005 and 2008.

Publication outlets
The 31 articles were published in 20 journals; the ratio of articles to journals was 1:1.55.The most frequently utilised journal was World Development with five articles, 18,19,20,21,22 followed by Development in Practice, 23,24,25 Journal of Health Management, 26,27,28 and Non-profit and Voluntary Sector Quarterly 29,30,31 with three articles each.The Journal of Development Studies published two articles. 32,33The least frequently utilised journals, with only one article each, were Accounting, Auditing and Accountability Journal, 34 Accounting, Organizations and Society, 35 AIDS Education and Prevention, 36 American Journal of Evaluation, 37 Development, 38 Evaluation, 39 Journal of Development and Social Transformation, 40 Political Perspectives, 41 Policy Studies Journal, 42 Progress in Development Studies, 43 Public Administration and Development, 44 Public Management Review, 45 Review of African Political Economy, 46 Texas International Law Journal, 47 and The Information Society. 48e 20 journals, in which the 31 articles were published, were clustered into six categories, based on core academic disciplines.The most preferred category was development studies journals with over half of the articles (18).There was medium preference for evaluation and accountability, and health journals (with 4 articles each).The least preferred were political journals (3 articles), and information technology and law journals (1 article each).
The journal cluster analysis by year shows that development studies journals have been utilised consistently over the years, with some growth since 2001.Evaluation and accountability journals have only been utilised since 2005, and the same trend is evident in the health journals.

Author collaboration and affiliation
The 31 articles were written by a total of 38 authors.For the purpose of this study, the number of authors per publication was a proxy indicator of research collaboration.Of the 31 articles, the majority (21) were sole authored, while only 10 were joint authored.Of the 10 joint-authored articles, 7 were written by two authors and the remaining 3 articles were written by three authors.Between 1997 and 2003, all articles (9) had sole authors; the first co-authored article was published in 2004.
In each article, authors reported their institutional affiliation at the time of publication.These institutions were categorised by country and are hereafter referred to as author location.
The resulting frequency of author locations is summarised in Table 1.The UK and USA were the most frequently reported author location, with nearly two-thirds of the total frequency.Overall, the likelihood that an author location was a developed country was four times greater than that of the location being a developing country.

Study location
As shown in Table 1, the most frequently reported study locations were India and Bangladesh.Other preferred study locations were the UK, USA and Uganda.
Overall, the majority of the studies were conducted in developing countries.The likelihood of the study location being a developing country was more than three times that of it being a developed country.Of the studies conducted in developing countries, half were conducted in Asia, just less than half in Africa and the remainder were in Latin America.

Author location compared to study location
We then compared findings of the study location with those of the author location to identify any relationship.The reported locations for each author and each study were categorised as developed country, developing country or both.
Of the 31 studies, 23 were written by authors in developed countries, 4 were written by authors in developing countries and 4 by authors in both developed and developing countries.Of the 31 studies, 6 study locations were in developed countries, 19 were in developing countries and 6 were in both developed and developing countries.
Of the 19 studies located in developing countries, 12 were reported by authors in developed countries, and the remaining 7 by authors in both developed and developing countries.Additionally, of the 13 authors who conducted studies in Africa, only 3 were affiliated to an institution in Africa.In contrast, the articles on the six studies located in developed countries were all authored by researchers in developed countries.

Study period
One challenge faced in undertaking this analysis was that 18 out of 31 articles did not explicitly provide information on the period of data collection.In 2009, effort was made to contact the study authors for further information.A response was received from the authors of only six of the articles.Hence, more than a third of all studies (12) did not specify the period of data collection.
Of the 19 articles with available information, the data collection period ranged from 3 to 36 months.Most of the studies collected data for 1 year or more.Further analysis was conducted on the time between the year of completion of data collection and the year the article was published.The time lag ranged from 1 to 6 years.The majority (15) of the studies experienced a time lag of 3 years or more, with only four studies having been published within 2 years of the completion of data collection.

Study topics
The topics examined in each study were extracted and analysed.There were nine main topic categories distinguished from the 31 studies.The most frequently mentioned topics were accountability (15); reporting (14); resource mobilisation (13); operational management (13); and monitoring and evaluation (10).The least examined topics were strategic location (3); needs assessment (5); institutional context (6) and organisational learning (6).
This finding should, however, be interpreted with caution, as the categories of topics identified, and the frequency with which they were reported, are products of the way in which the studies defined their topics.For instance, some studies used the term 'governance' while discussing 'accountability' issues.Further, there seems to be an overlap between various practices, as some topics were examined under the banner of others.Most commonly, accountability emerged as an umbrella term that incorporates aspects of governance, financial reporting, monitoring and evaluation.Additionally, organisational learning involves the processes of monitoring, evaluation and reflection whereby lessons are learnt from experiences.This finding reflects an overlap and interconnection between the different organisational processes and organisational performance assessment processes.This finding further shows that the field of organisational performance is multidimensional, and relevant to a broad range of management themes.

Study methods
A majority (26) of the 31 studies reported primary data, whilst only a few examined secondary data.The studies utilised both qualitative methods (such as interviews, focus group discussions and workshops) and quantitative methods (particularly Note: Articles that reported study locations as 'Africa' or as 'developing countries' and those for which a location was not given have been omitted.*, The reported frequency is higher than the number of articles because some articles had more than one author, some authors were affiliated to institutions in more than one country or some studies were conducted in more than one country. questionnaire surveys).Similar variation was observed in the sample sizes, which ranged from the examination of one organisation to nationwide surveys of NGOs.

Sources of information
Figure 2 shows the frequency with which each source of data was reported.The most preferred sources of data were NGO staff, documents and records, and donor agencies.
The least utilised sources were beneficiaries, consultants (including technical assistance providers, accounting bodies and external experts), governments, academics and others (including labour organisations and the press).

Discussion
This study informs policy directions and future research on factors influencing NGO performance.The initial large number of citations (14 469) on NGOs shows that a large volume of literature exists on NGOs, but is contained in books, organisational reports, newspaper and newsletter articles, and other types of grey literature.Academic literature is therefore not a major outlet of publishing or source of reading for most development practitioners.
Despite the large growth of interest in NGOs and the large body of NGO publications, there are a relatively small number of academic peer-reviewed journal articles that examine factors influencing NGO performance.These findings substantiate previous arguments that the organisation and management of NGOs has received relatively little attention from researchers, 49 and that the subject of NGOs has not yet entered the academic mainstream and hence the overall state of knowledge remains somewhat underdeveloped. 5The low academic research volume may be attributed to several factors.Firstly, the fields of NGO management and research are relatively new, 49 and measuring NGO performance continues to be relatively underdeveloped. 6Secondly, many evaluations and studies are not published -numerous organisational reviews are kept confidential because there is a bias towards publishing only positive results, with failures concealed because of various sensitivities and fears (particularly the fear of revocation of funding).There is also a bias towards undertaking reviews and publishing results from well-resourced organisations. 4,50Thirdly, this study indicates that the lengthy, intensive and costly nature of the research process may mean that such studies are difficult to undertake in low-resourced NGO settings.One proxy indicator is the long period of data collection -most studies took more than 1 year, with some taking up to 3 years.However, these figures need to be treated with caution, as the duration is a product of the way studies reported the period of data collection and of the methodology of the study.Another proxy is the time lag between completion of data collection and publication of an article, as a majority of the articles had a time lag of 3 years or more.This length suggests that data capture, analysis, verification, writing, peer-review and acceptance of an article for publication are extremely intensive, time-consuming processes, which may reduce the likelihood of studies being published in this field.Nonetheless, this time lag is not atypical of research work in other areas which culminates in the publication of a journal article.
Interest in factors influencing NGO performance has found a home in a wide array of journals.This observation is consistent with sentiments by Edwards and Fowler 3 that NGO literature is widely dispersed and locating it requires long and difficult searches -as was the experience of the researchers in this study.The same observation may further confirm that NGO performance is a multidisciplinary field continually drawing ideas and techniques from other fields, 49 particularly from international development, management, health, politics and law.
Location analysis by author and study reveals that there is an international scope in researching NGO performance.Authors were located and the studies conducted in both developing and developed countries covering the continents of Africa, Asia, Australia and Oceania, Europe, North America, and South America.That there were comparatively few studies in Latin America is attributed to the fact that only English-language publications were included.
Further analysis revealed that most authors were affiliated to institutions in developed countries, although the majority of studies examined NGOs in developing countries.Author and study location analysis revealed that most studies in developing countries were conducted by researchers in developed countries, whilst all studies in developed countries were conducted by researchers in developed countries.Whilst there is rhetoric of 'North-South' and even 'South-South' collaboration and participation in international development, 51,52,53 this collaboration is not evident in the research on understanding NGO performance.
This striking finding raises further questions.The first is whether this trend could be attributed to low capacity in developing countries.The predominance of author locations being in developed countries may suggest that these institutions have more established resources, reputations and capacities to examine NGO performance than those of institutions in developing countries, and hence scholars interested in this field are affiliated with these institutions.According to Mishra et al. 54  Based on the findings of this study, several areas of future research can be identified.Firstly, future research should explore the wide range of factors that influence NGO performance, including the institutional context and influences of the external environment, 56 along with the strategic location of NGOs.These factors may provide evidence for policy and funding allocation effectiveness to improve beneficiaries' access to essential resources. 2condly, although providing details of the period of data collection is a usual requirement in research, less than half (13 out of 31) of the studies explicitly met this requirement.Some studies did not discuss the methods used.Future research in the area of understanding NGO performance would benefit from an explicitly reported methodology section.
Thirdly, future research should engage more rigorously with the opinions and needs of beneficiaries.Analysis of the sources of data distinguished eight categories, reflecting the trend that researchers gather data from various sources, confirming that a wide range of stakeholders are involved in varying magnitudes in assessing NGO performance.However, further analysis showed a marked preference towards gathering data from NGO staff, particularly from senior-level and middle-level managers including directors and executives, programme coordinators and programme staff.Few studies endeavoured to source information from NGO field workers.Comparatively few studies gathered data from beneficiaries, which suggests that the rhetoric of beneficiary participation and engagement seems not to have been effectively implemented in research projects, and may be a proxy reflection of the reality in intervention practices.
Of the two forms of participation -in implementation and in decision-making -beneficiaries are often involved in implementing activities based on decisions taken in other fora.Beneficiaries are important stakeholders who should be involved from the design to evaluation, and back again to reformulation of development interventions. 57,58Furthermore, incorporating their voices into studies generates data that would otherwise be overlooked and can make the realities and experiences of beneficiaries more prominent. 59,23nally, we turn to focus on the African continent.Strikingly, only 3 of 13 authors who conducted studies in Africa were affiliated to an institution in Africa, namely in Cameroon, South Africa and Tanzania.These 3 authors represent less than a tenth of all the 38 authors of the 31 articles.This result reveals the need to address the low number of published researchers in Africa in this field, and the crucial need to nurture African researchers.Along with this need, the scarcity of partnerships with African-based researchers in the field of NGO performance research needs to be further addressed.

Strengths and limitations
This study makes a unique contribution to the social sciences, as relatively few systematic reviews have been employed as a methodology in social science research, compared to other sciences.The study makes a contribution to research in NGO performance, and provides information on journals that publish articles on understanding NGO performance.This study also makes recommendations for areas of future research that policymakers and donors may require to inform future funding decisions.
However, there are limitations that should be taken into consideration.The search was restricted to online academic articles, written in English, specifically focused on factors influencing NGO performance.The search criteria thus excluded journal articles in other languages and in nonelectronic journals, and other published but 'grey' literature.These publications could have possibly contained relevant data that may have made significant contributions to our understanding of development NGO performance.

Conclusions
This study described an international structured literature review, and explored the salient characteristics of NGO performance research.We have demonstrated that understanding NGO performance is a multifaceted arena in which many issues are explored by academics, NGO practitioners, donors, governments and policymakers.Notwithstanding the modest number of studies (31) that fully met the inclusion criteria, emergent evidence supports three key conclusions and recommendations with research, practice and policy implications.
Firstly, from 1996 to 2008 there was steady growth in the number of journal articles reporting factors influencing NGO performance.However, given the small number of studies, there is a need to increase published research, particularly research examining the facilitating and constraining factors influencing NGO performance.These studies would benefit from the inclusion of an explicit and more robust methodology section to enhance further meta-analysis and comparison studies.Secondly, there is a need to increase the practice of hearing beneficiaries' voices, not only in assessing beneficiaries' needs, but also in gathering their perspective regarding organisational performance.Finally, this study found plausible evidence of a relatively small number of published researchers based in Africa in the field of NGO performance assessment.Additionally, little academic research on NGO performance is undertaken in partnership with researchers in Africa.These findings reveal the need to advocate for policymakers in academic, donor, government and development institutions to intensively identify and nurture researchers in Africa in the field of development and performance management.

FIGURE 1 :
FIGURE 1: The number of articles published between 1996 and 2008 on factors influencing the performance of NGOs.

23 FIGURE 2 :
FIGURE 2: Frequency of sources of data in articles published between 1996 and 2008 on factors influencing the performance of NGOs.

TABLE 1 :
Frequency of study location and locations of authors of articles published between 1996 and 2008 on factors influencing the performance of NGOs.
55India has many institutions to prepare management professionals in the private corporate sector, but none in the NGO sector, hence scholarship in development management of NGOs has lagged far behind practice.There may also be additional challenges for developing country authors to break into discourses, such as engaging in the academic style of writing, prejudice by journal editors and publishing networks.Another question is whether there is a trend of 'Northern' institutions examining 'Southern' institutions, particularly as large amounts of development funding originate from 'Northern' countries.Wallace and Chapman 52 argue that while UK NGOs and donors state commitment to downward accountability and promotion of local ownership and control of development, the policies and procedures of funding disbursement and accounting ensures that upward accountability dominates, which is part of a wider problem of donor domination of recipients.According to Muchungunzi and Milne55, the 'South' always has to account to the 'North' and not vice versa; donor money and expertise from the 'North' are more valued than are labour and expertise from the 'South'.