| Slide1 : Evaluating public RTD interventions:
A performance audit perspective from the EU
Hendrik Fehr, Gareth Roberts
European Court of Auditors
American Evaluation Society
Portland, 3 November 2006 |
| to discuss.... : to discuss.... Background
the European Court of Auditors
the European Union budget
What do we mean ‘performance audit’?
Current audit: The European Commission’s system for evaluating the impact of RTD interventions
why this audit, why now?
audit methodology
issues arising to date |
| European Court of Auditors : European Court of Auditors Independent institution – external auditor of European Union
Reports sent to European Parliament and European Council of Ministers
25 Members, based in Luxembourg, organised into 5 groups, covering:
agricultural policies
structural and internal policies
external actions
own resources, banking, admin
co-ordination, quality assurance
See www.eca.europa.eu |
| European Union Budget (i) : European Union Budget (i) |
| European Union Budget (ii) : European Union Budget (ii) Research
6bn euro (approx) |
| Framework Programmes for RTD (i) : Framework Programmes for RTD (i) 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 FP4 FP5 FP6
(avg 5bn pa) FP7
(avg 7bn pa) Run-in: developing new programme Shadow: implementation to the end |
| Framework Programmes for RTD (iii) : Framework Programmes for RTD (iii) directly managed by the European Commission
mainly extra-mural
multi-partner consortia (covering more than 1 EU Member State)
governed by contracts / grant agreements
several instruments / types of projects
networks of excellence
integrated projects
specific targeted research projects
specific support actions
etc |
| What do we mean by ‘performance audit’? : What do we mean by ‘performance audit’? Court’s mandate derived from treaty
provide a statement of assurance as to the reliability of the accounts and the legality and regularity of the underlying transactions,
examine whether the financial management has been sound
submit observations on specific questions and deliver opinions Performance audit:
management of research under FP5
audit of effectiveness of instruments
audit of evaluation system Financial audits:
annual report / opinion Opinions:
Rules of Participation for FP7 |
| What do we mean by ‘performance audit’? : What do we mean by ‘performance audit’? According to the Court’s audit manual, a performance audit is
an audit of sound financial management, namely of the economy, efficiency and effectiveness.
to put it simply, examining
whether the right things are being done, and
whether things are being in the right way
|
| The focus of the current audit : Overall audit question
Does the European Commission have an adequate system for evaluating and assessing the impact of indirect RTD actions? Sub question 1:
Are there clear relationships between the policy objectives of the framework programmes and the criteria and indicators used to select, evaluate and assess the impact of RTD projects? Sub question 2:
Are evaluation activities carried out in a comprehensive, coherent and independent manner and are relevant and useful results communicated on a timely basis, and are these results taken into consideration for future policy decisions? The focus of the current audit |
| Why this audit (now)? : Why this audit (now)? because evaluation is relevant in the audit world.....
for transparency and accountability
for good governance and improving programme management
for informing policy decisions
assessed as an issue of particular risk because of...
the inherent difficulty in assessing the impact of public R&D interventions given the complexity of relationships between inputs, results and impacts, and,
the increasing levels of European Community funding
the concerns expressed in the Court’s earlier work about the information available about the outputs, outcomes and impacts of Community-funded research actions
to contribute to the development of an improved evaluation system for the future |
| the audit methodology : the audit methodology Work focused on the Commission:
analysis of existing evaluation system
analysis of objectives and performance criteria
analysis of evaluation reports and
methodologies deployed Work aimed at understanding good practice:
benchmarking visits to USA and Canada
literature review
stakeholder consultation within the Member
States Work to validate credibility of findings
expert / stakeholder focus groups
contradictory procedure with the
Commission |
| Issues arising from work to date....... : legal and institutional framework intervention logic etc evaluation findings implementing RTD evaluation Issues arising from work to date....... |
| Legal and institutional framework : Legal and institutional framework What are the main elements of evaluation governance? and what impact do these have?, eg:
Legal framework
Reporting results and link with budgetary processes
Internal evaluation and other standards
Quality assurance of the evaluation system
Who is responsible for monitoring the functioning of the evaluation system?
What influence do they have in practice?
Is a distinction between monitoring and evaluation |
| Intervention logic etc : Intervention logic etc Is intervention logic clear from the outset?
Are objectives SMART?
How are performance indicators defined and targets set? How is performance against these targets monitored? |
| Implementing RTD evaluation : Implementing RTD evaluation How far is programme evaluation taken as a important priority by top management?
Is a robust evaluation strategy in place? If so, how well does it
consider how to deal with inherent complexity
explain intervention logic
take account of the role of evaluation in the policy cycle
strike a judicious balance between in-house capacities and external expertise
cover the portfolio of issues to be evaluated , program level challenges
deploy a mix of quantitative and qualitative methodologies etc
Are resources adequate for evaluation needs?
human and financial resources
data collection / IT systems |
| Evaluation findings : Evaluation findings What kind of information does the evaluation system produce?
To whom is it disseminated?
Do stakeholders consider it to be useful information? |
| Comparing ‘performance audit’ and ‘evaluation’ : Comparing ‘performance audit’ and ‘evaluation’ Similarities:
Methodologies and techniques Differences:
Institutional context
Evaluee relationships
Reports Do you welcome such an initiative from an audit institution ? |
| Contact details : Contact details Gareth Roberts
Auditor
Tel: +352 4398 45724
gareth.roberts@eca.europa.eu Hendrik Fehr
Head of Division
Tel: +352 4398 45503
hendrik.fehr@eca.europa.eu Internal Policies (including Research) Division
European Court of Auditors
12 rue Alcide de Gasperi
L-1615 Luxembourg
http://www.eca.europa.eu |