Show
The appropriate timing and type of evaluation required to support continuous improvement, accountability and decision-making needs to be determined on a case-by-case basis to ensure the overall approach is fit for purpose. It is not feasible, cost effective or appropriate to fully evaluate all government activities and programs. The cost of evaluation must be balanced against the risk of not evaluating, noting that sometimes performance monitoring by itself will be sufficient to meet the performance reporting requirements under the Public Governance, Performance and Accountability Act 2013.
In some cases, well-designed data collection and performance monitoring established during the design phase of a new or amended program or activity can help to refine it over time. In other cases, evaluation will need to be more comprehensive to assess whether an activity or program is appropriate, effective and/or efficient. Taking a strategic, risk-based approach can help to:
Identifying and prioritising if and when an evaluation is requiredWhat to evaluate, when to evaluate, and the type of evaluation needed in a particular circumstance depends on a range of factors. A strategic, risk-based approach can help to determine when and how evaluations are selected, prioritised and scaled based on the value, impact and risk profile of a particular government activity or program. The topic, scope and size of each proposed evaluation will vary, so these issues will not apply in the same way or have the same relative importance in all evaluations.
Different types of evaluationDifferent types of evaluations draw on a range of methods and tools to measure and assess performance. These approaches can be organised in different ways, with common distinctions made between formative and summative evaluations. Rapid approaches are also being used in the public sector to meet the information needs of decision-makers in uncertain, resource-constrained environments. Questions about the level of need, policy design and implementation/process improvements are usually best answered during the early design and implementation/delivery stages of a program or activity (these evaluation approaches and techniques are generally referred to as formative evaluation). Questions about program outcomes and impacts are usually best answered near or at the end of the policy/program or after it has matured (these evaluation approaches and techniques are generally referred to as summative evaluation). What type of evaluation is best suited in a particular situation depends on a combination of:
It is important to select tools and approaches that are fit for purpose based on the specific program or activity and the purpose of the evaluation. See How to evaluate for information about things to consider at each stage of planning and conducting an evaluation, including defining its purpose, areas of focus and design. Evaluation at different stages of the policy cycleEvaluations can be used to support continuous improvement, risk management, accountability and decision-making at various stages in the policy cycle. One way of doing this is through a program logic model. The purpose of a program logic model is to show how a program works and to draw out the relationships between resources, activities and outcomes. While this program logic is presented in a linear fashion, in practice there will be continual feedback loops where evaluation findings and performance information help to inform continuous improvements in policy development and program design.  Broadly, the stages when an evaluation is conducted include:
Questions: Will it work? To what extent is the need being met? What can be done to address this need? What is the current state? Areas of focus:
Program characteristics: Program is innovative and in development – exploring, creating, emerging. Type of evaluation/other terms: Implementation readiness assessment, formative evaluation. Commonwealth requirements: Before an activity or program is implemented, entities need to plan how an activity or program will be evaluated in order to:
Risk and assurance review processes, including appropriate use of the Risk Potential Assessment Tool (RPAT) and the two-pass capital works and ICT reviews, are also required for proposals that meet certain risk and materiality thresholds.
Questions: Is it working? Is the program or activity operating as planned? What can be learned? Areas of focus: Feedback to support early course correction, and provide insight into the program or activity’s operations, implementation and service delivery. Program characteristics: Program is forming and under refinement – improving, enhancing, standardising. Types of evaluation/other terms:
Commonwealth requirements: During the delivery of a program or activity, Commonwealth entities have an ongoing legislative requirement to measure, assess, and report on their performance under the Commonwealth Performance Framework. At this stage of the policy cycle, effective, fit for purpose evaluation approaches can be used:
Questions: Did it work? Is the program or activity achieving its objectives? What has the impact been? What are the benefits relative to the costs? Areas of focus:
Program characteristics: Program is stabilising, well established, mature and builds on data collected in earlier stages. Type of evaluation/other terms: Outcome evaluation, impact evaluation, summative evaluation, ex-post, theory of change evaluation, economic evaluation. Commonwealth requirements: Commonwealth entities have an ongoing legislative requirement to measure, assess, and report on their performance under the Commonwealth Performance Framework. Once a program or activity has matured and been in operation for some time, effective, fit for purpose evaluation approaches can be used:
Questions: What evidence is available? What action is required? What caveats do I need to put on the findings? Areas of focus: An approach designed to quickly and systematically conduct an evaluation when time or resources are limited. Program characteristics: The common feature of this approach is the expedited implementation timeframes, which generally range from 10 days to 6 months, for situations where a short-term or immediate outcome is expected, or a quick decision is required. These methods have been used in multiple settings including public health, emergency management, international development and agriculture as a way to deliver program evaluation findings quickly to inform decision making, for example in a public health crisis where improvement in infection control is expected to happen within a short timeframe. Types of evaluation/other terms: Real time evaluations, rapid feedback evaluation, rapid evaluation methods, rapid-cycle evaluation, rapid appraisal. Commonwealth requirements: Decision-makers (i.e. government, responsible ministers, accountable authorities, and senior managers) will determine when a rapid evaluation is required to support urgent action, decision-making and accountability. For more information, see Rapid Evaluation | Better Evaluation -- NAVIGATING THE EVALUATION TOOLKIT -- Go to Evaluation in the Commonwealth Page 2
A critical step for turning your evaluation into meaningful information that supports continuous improvement, accountability and decision-making is summarising and discussing the main findings. This usually takes the form of an evaluation report, but other products may also be required to meet the needs of different stakeholders (e.g. a plain English summary document, different language versions depending on the stakeholder cohort, or a presentation for participants, staff or delivery partners).
The report findings should answer the evaluation questions established during the planning phase, identify any implementation challenges or limitations, and help decision-makers to understand whether the program or activity is on track and meeting its objectives.
An evaluation report typically makes constructive, actionable findings/recommendations and provide lessons learned to support continuous improvement. It is important that the report meets the needs of different and diverse stakeholder groups to have maximum impact.
What do the results tell you?
How will you turn the evaluation results into meaningful information for decision-makers?
How will you share the evaluation findings?
Evaluation ReportAn evaluation report will typically include:
Different groups of stakeholders may want to see the results of the evaluation in different forms. As you conclude your evaluation, it is worthwhile checking in again with all the stakeholder groups to clarify their reporting requirements are (i.e. what needs to be reported and when). Reporting timelines often present a major constraint on the evaluation. In particular, the need to report findings in time to inform funding and resourcing decisions for the next phase of a program often means that reports are needed before impacts can be fully observed. In these situations, it will be necessary to report on interim outcomes, and to present any research evidence that shows how these are important predictors or pre-requisites to the final impacts. -- NAVIGATING THE EVALUATION TOOLKIT -- Go to Evaluation in the Commonwealth Page 3Evaluations generate rich, evidence-based insights that contribute to good public administration. Irrespective of whether evaluation findings are positive, negative or neutral, evaluation reports are not intended to just “sit on the shelf” or be seen as a “tick a box” activity – they are designed to be used constructively to support continuous improvement. An implementation plan, endorsed by senior executives, will help to implement evaluation findings and prioritise actions to be taken to improve programs and services, enhance design and delivery mechanisms, and ultimately improve outcomes. A communication plan will also help to ensure findings are shared in the most appropriate way with stakeholders, participants, and a broader audience. This allows end users of your program or activity to benefit from continuous improvement, and ensures the evidence base can be used and expanded to help similar programs improve their outcomes. Evaluation findings should be transparent by default unless there are appropriate reasons for not releasing information publicly. This supports accountability, continuous improvement, and helps to embed a culture of evaluation across the Commonwealth.
How should my entity use the results of the evaluation?
How does my evaluation support learning across the Commonwealth?
Does my evaluation help continuous improvement of other programs and activities in my entity?
Review evaluation objectives and implement improvementsOnce you have delivered a well planned and executed evaluation, it is important to ensure the results are used and the learnings are disseminated. Prioritise the improvements: Often the evaluation will have a number of findings/recommendations. In order to maximise impact, it’s important to prioritise those recommendations that can be implemented immediately, and those that can wait. Develop an implementation plan that is signed off by the executive in the entity: Developing an implementation plan is a good way to carefully plan the changes needed in the program or activity within any resource constraints. This plan would involve:
Have a communications plan: A Communication Plan outlines the strategies that will be used to communicate the results of the evaluation. The plan needs to describe which results will be communicated, how they will be communicated, and to whom they will be communicated. It is important to consider the different aspects and techniques for discussing the evaluation results with stakeholders and a wider audience to support continuous learning across the Commonwealth.[1] Monitor the implementation of evaluation recommendations: Program managers are responsible for implementing the agreed recommendations of their program or activity. Share the learnings: The benefits of properly planned and executed evaluations are considerable, for the government, the Commonwealth and, importantly, for Australians – they generate rich, evidence-based insights that help guide the allocation of public resources, improve the design and implementation of programs, and deliver better services. Sharing the tips and tricks you have learned through your evaluation across the Commonwealth will help to build evaluation capability. It also allows the evidence base to be used and expanded to help similar programs and activities to improve their delivery and outcomes.
If you have an evaluation case study that you would like to share, please contact us at . -- NAVIGATING THE EVALUATION TOOLKIT -- Go to Evaluation in the Commonwealth Page 4
Whenever you design, run or evaluate a Commonwealth program or activity, it always occurs in a particular context. It is important to understand your entity's operating context, purposes, key activities and current performance framework – outlined in your Corporate Plan – in order to decide: It is important to consider how the objectives of a specific program or activity contribute to your entity achieving its purpose/s, and whether you have all the performance information you need to assess how well those objectives are being met. What factors in my operating context need to be considered? What are the purposes of your entity? What are the objectives of your program or activity? What factors (environmental, risk, capability, and partnerships) must be managed to achieve your purpose/s? How does your program or activity contribute to achieving your entity’s purpose/s? How do you currently forecast, measure and assess the performance of your program or activity against its objectives? Has an evaluation been included in a new policy proposal, or in your entity’s corporate plan? Is there a government direction or legislative requirement to do an evaluation? What is the level of support and capability for doing an evaluation in your entity? Does your entity have an evaluation strategy or an evaluation unit to guide you? How can I identify and prioritise what should be evaluated? What matters most to the people that matter? Where is the activity or program in the policy development cycle? Can an evaluation support, complement or enhance your regular performance monitoring and reporting activities? Can evaluation methods and tools be used to improve the quality of reporting in corporate plans and annual performance statements? Note all Commonwealth entities are required to forecast, measure, assess and report on their key activities throughout the reporting cycle – where would the proposed evaluation fit in? What is the main purpose of my evaluation, who is the audience and how will the evaluation be managed? Understanding your operating contextUnderstanding your entity's current operating context, and where a specific program or activity fits in, is critical for deciding what should be evaluated and when. Working with stakeholders can help to:
Commonwealth entities are required to publish a discussion about their operating context in their corporate plan. This discussion should provide a clear understanding of how, individually and collectively, things like: capability levels; risk oversight and management; and cooperation with others contribute to an entity achieving its purposes over the four year period covered by each plan. Reviewing your entity's corporate plan can help you develop an understanding of where a specific program or activity fits within your operating context, and how an evaluation might help to strengthen performance reporting and accountability within your entity. The performance information collected for your specific program or activity may be more detailed than the high-level performance information reported in your entity’s Corporate Plan or Portfolio Budget Statement. Ideally, you still should be able to identify which key activity it relates to and the contribution it makes towards meeting your entity's purpose/s. Findings from past evaluations can also help to inform a more mature discussion about program performance and your entity’s operating context in future corporate plans. This ensures that there is continuous feedback loop where evaluation both informs, and is informed by, your entity's routine performance monitoring and reporting. -- NAVIGATING THE EVALUATION TOOLKIT -- Go to Evaluation in the Commonwealth |
What are the 3 rapid evaluation questions?
Postagens relacionadas
direito autoral © 2024 estudarpara Inc.
