Sample ToR for evaluation of Central Sector Schemes - reg.

No. 66(59)/PFC-II/2018 Ministry of Finance Government of India Department of Expenditure kkk North Block, New Delhi Dated the oc ay 2025 OFFICE MEMORANDUM Subject: Sample ToR for evaluation of Central Sector Schemes- reg. Reference is invited to the Budget Speech of the Finance Minister in 2016 wherein he had outlined the policy framework as regards the schemes being implemented by Government of India. Para 110 of the Budget Speech stated that “To improve the quality of Government expenditure, every new scheme being sanctioned by the Government will have a sunset date and outcome review”. Pursuant to this announcement, vide O.M. No. 42(02)/PF- 11/2014 dated 23.2.2017, DoE had informed Ministries/Departments that after the end of the 12 Five Year Plan, “for aligning the schemes with financial resource cycle of Central and State Government, schemes will be made co-terminus with the Finance Commission Cycle, the first such being the remaining period of the 14k Finance Commission period”. It is further stated that approval for continuation of scheme must be sought based on the outcome review showing that though scheme has been effective in achieving its objectives, still there is a need to continue the scheme in view of its mandate and performance. Depending upon the outcome review, scheme can continue in its existing form or with necessary modifications. The Cabinet Secretary vide his D.O. Letter No. 1/50/1/2016-Cab. Dated 19.12.2017 (copy enclosed) had emphasized the need for evaluation of schemes. All such proposals for continuation of schemes needed to be preceded by an appropriate evaluation and necessary restructuring carried out in them so as to achieve the desired results. 2. DMEO, NITI Aayog is conducting evaluation of the Centrally Sponsored Schemes. DoE after consulting DMEO, NITI Aayog had issued instructions for evaluation of Central Sector Schemes vide its OM of even no. dated 22.5.2019 enclosing sample Terms of Reference. Ministries/ Departments were advised to use the Terms of Reference enclosed alongwith ibid O.M. for evaluation of Central Sector Scheme being implemented by the respective Ministries/Departments and, in case of modifications to seek guidance from NITI Aayog. 3: As the terminal year of the current Finance Commission Cycle has started, it is imperative that Ministries/Departments make an assessment of Central Sector Schemes they wish to continue over the next FinanceCommission Cycle. In terms of the Policy mandate of the Government announced in 2016 Budget Speech and followed by the instructions contained in the D.O. Letter of Cabinet Secretary an evaluation of the Central Sector Schemes which are intended to be continued needs to be carried out. Central Sector Schemes for which evaluation is being carried out/has been done by NITI Aayog (list of such schemes is at Appendix-I) need not be re-evaluated. 4. In’ this regard, please find enclosed final Standard Terms of Reference (ToR) Template for evaluation of Central Sector Schemes. The first part of the template outlines the structure of the report and the second part is the explanatory memorandum. With the issue of this template which is generic in nature, individual vetting of ToR for each Central Sector scheme by NITI Aayog will not be required. However, in the opinion of the Ministry/Department, if any particular scheme so demands, the Ministry/Department may like to approach NITI Aayog for specific addendum or revision in this template. 5. This issues with the approval of Secretary (Expenditure). (Hema Jaiswal) Dy. Director General (PFC-II) Financial Advisors All Ministries/Departments of Government of India Copy to: Director General, Development Monitoring and Evaluation Office (DMEO), NITI Aayog, New Delhi.Template for Evaluation of Central Sector Scheme: [Scheme Name] Name of Department /Name of Ministry 1. EXECUTIVE SUMMARY 2. OVERVIEW OF THE SCHEME 2.1 Background of the Scheme a) Brief write up on the scheme including objectives, Implementation Mechanism, scheme architecture/ design b) Name of Sub-schemes/ components c) Year of commencement of scheme d) Present status with coverage of scheme (operational/ non-operational) e) Sustainable Development Goals (SDG) Served f) Alignment with Viksit Bharat 2047 Vision g) Fund Flow mechanism (National and sub-national level) 2.2 Budgetary allocation and expenditure pattern of the scheme Sub- {Year 1] [Year 2] [Year 3] [Year 4] [Year 5] sche me/ Comp] BE | RE | Actual} BE | RE |Actua| BE| RE |Actual| BE |RE/Actua| BE | RE |Actu onent l 1 al 2.3 Summary of past evaluation since inception of scheme Year of Agency hired for Recommendations Recommendations made Evaluation Evaluation made and accepted but not accepted 3. METHODOLOGY 3.1 Sample size and sample collection process- The approach should be to derive a sample size based on some key output/outcome parameter of the scheme. The States/Districts/Villages & UCBs could be graded in terms of performance of the states in these indicators and accordingly representation from each such Strata be finalized systematically. While doing so, regional representation in terms of Zones as per NSSO (consisting of States/UTs) and aspirational districts/blocks may also be kept in mind. ie3.2 Baseline data on Key Performance Index (KPIs): Baseline data on select KPIs (15 to 25 central to measure the effectiveness of the scheme in terms of its meeting the scheme objective to be collected: - 3.2 (a) For New Scheme: Baseline data to be collected through a baseline survey from the field. This could be done anytime during the commencement of the scheme to six months of it being in operation. This data and accompanying report are to be kept safely by the M/Ds. This would be followed by a mid-line survey, ideally after 2-3 years of its operation. 3.2 (b) For Existing Scheme: Synthetic baseline data may be created using secondary and administrative data to which these KPIs are to be compared. In any case, the current evaluation data has to be stored properly to be used in the future as an anchor dataset to carry out proper analysis of the ongoing schemes. 3.3 Evaluation Tools used: Various evaluation tools can be used in the primary survey such as Household questionnaires, Key Informant Interviews, Focused Group Discussions, etc. This would depend on the focus and orientation of the scheme. 3.4 Data collection tools: Computer Assisted Personal Interview (CAPI) tools to be adopted. 4. OBJECTIVE OF THE STUDY 4.1 Effectiveness of the scheme in terms of achieving its intended objectives as per the scheme guidelines (To be analysed on the Output /Outcome/Impact indicators) 4.2 Additional parameters a) Awareness and Coverage of Scheme-uptake by beneficiaries Sate District Urban | Rural Urban Rural | Male |Female| Male | Female| Male |Female| Male | Female sc/st | sc/sT |Sc/st | Sc/sT [SC/ST | SC/ST | SC/ST | SC/ST b) Assessment of implementation process and fund flow mechanism c) Training/ Capacity building of administrators/ facilitators, IEC activities ete. d) Asset/ Service creation, if any, assess adequacy and quality and & maintenance plan/recurring expenditure associated with it igh rale) Benefits (Individual, community) f) Convergence with scheme of own Ministry/ Department or of other Ministry/ Department and with the scheme of States/UTs. 4.3 Gaps in achievement of outcomes 4.4 Key Bottlenecks & Challenges 4.5 Input Use Efficiency 4.6 Best Practices/Case Studies 4.7 Most Significant Change (due to the scheme/intervention) 5. OBSERVATIONS AND RECOMMENDATIONS 5.1 Thematic Assessment i. Improving value for money through identification of avoidable duplication and overlaps between different schemes ii. Assessing the continued relevance and their effectiveness and efficiency in achieving the intended objectives. iii. Reduction in avoidable overhead expenditure on consultants, programme management, administration etc. 5.2 Externalities 6. CONCLUSION 6.1 Issues & challenges (data must attempt to gather information on, but not limited to, the following) Has the intervention caused a significant change in the lives of the intended beneficiaries? How did the intervention cause higher-level effects (such as changes in norms or systems)? Did all the intended target groups, including the most disadvantaged and vulnerable, benefit equally from the intervention? Is the intervention transformative — does it create enduring changes in norms — including gender norms — and systems, whether intended or not? Is the intervention leading to other changes, including “scalable” or “replicable” results? How will the intervention contribute to changing society for the better? 6.2 Vision for the futureEXPLNATORY MEMORANDUM 2.1 (a) Latest guidelines of the scheme may also be annexed. 2.1 (b) Sub-schemes/components which have separate budgetary allocations either through budget line or through Detailed Demand for Grants (DDG) may also be listed along with component-wise budget allocation. 2.1 (d) Details of number of states/districts/villages where the scheme is functional may be included. 2.1 (e) & (f) May be linked with objective of the scheme. 2.2 If DDG makes allocations for certain items which are not shown as sub- schemes/ components, the same may be separately indicated along with allocation in the table. 3.1 Complete list of states as classified by NSSO may be seen on its website. 4.1 Output/ Outcome Indicators as per the objectives of the scheme be specified for proper assessment of the scheme. Indicators as spelt out in SFC/EFC memorandum during appraisal of the scheme may also be utilized. Output/ Outcome/Impact Indicators (numbers or percentage) must be compared with base year value at all time points (periodicity) as per monitoring mechanism framework, also defined in SFC/EFC Memorandums 4.2 (a) Tabulated information up to Tehsil/ Block Level and if possible, up to village level may be provided. 4.2 (b) focus should be on clarity of instructions, availability of scheme or programme guidelines, clear definition of roles and responsibilities of functionaries and the number and nature of clarifications/ additional instructions issued w.r.t. scheme guidelines. Details about training (PFMS/EAT module, scheme’s portal or any other) with number of interventions and levels at which these interventions were carried out may be included. 4.3 (c) Details about Stakeholders/ Beneficiaries, details of campaigns, media, frequency, feedback etc. may be included. 4.2 (f) In primary survey, a facility checklist may be included to ascertain the quality and usability of such assets created under the scheme. Details about nature and type of convergence with national and state level schemes may be allDetails about apparatus- manpower office transport etc. may be included. If there is no convergence, NIL may be recorded. 4.3 These gaps could be attributed to absence of interventions/ non- performance of existing interventions. 4.4 Focus may be of Financial, Administrative, Project Management and any other Key Bottlenecks & Challenges. 4.5 Details of (a) requirement of funds as indicated in EFC/ SFC in relation to actual allocation of funds including timelines of release (b) requirement and allotment of manpower in implementation of scheme/Programme at various levels (PMU/Central/State) (c) Involvement of private players, volunteers, non- governmental organizations and local community etc. in the scheme may be provided. 4.6 Description of the Best Practices and Case Studies- Since evaluation is to record both what is working and what is not, therefore, the best practices should be recorded in detail, facilitating replication. 4.7 Most Significant Change: This information is to be gathered from primary survey w.r.t what one thing that the scheme has been able to achieve in the most significant way. 5.1 Thematic assessment should focus on Accountability, Transparency, Employment generation (direct/indirect), Climate change and sustainability, Role of TSP/SCSP, Use of IT, Behavioral change in stakeholders/beneficiary, R&D, Role, functions, involvement/ support of State govts. Evaluation of public interventions and policies needs to recognize the scarcity of resources, fiscal affordability and the ‘zero sum game’ that plays out in resource allocation. In a developing country like ours which faces multiple social, economic and environmental challenges, availability of public resources amongst competing demands will always be limited. Priority setting and resource allocation are vital questions to be addressed, and evaluation needs to assist in this. While interventions may be chosen to maximize specific objectives, they all are constrained with respect to practical and budgetary issues. Hence, evaluation recommendations should not routinely assume enhanced funding. If felt essential, alternative non-base case recommendations may be made separately. Cross cutting themes can be assessed both through secondary data as well as primary. While conducting meta-analysis of existing reports, the evaluator Poshould actively review the cross-cutting themes. The primary data for cross cutting themes will be elicited through specific questions and responses during the key informant interviews and beneficiary surveys. For example, use of IT in scheme implementation, fund flow, monitoring and evaluation can be assessed from interaction with concerned ministries/ departments as well as states officials. Similarly, gender mainstreaming can be assessed by introducing specific questions on changes in knowledge, attitude and practices pertaining to gender equality, attributable to the CS intervention at hand, through household surveys. 5.2 Details best practices, innovations or scheme/ projects where best practices were replicated may be provided. 6.3 It is essential to highlight the importance of recommendations for the scheme. The evaluation agency may provide recommendations for the scheme in any of the following categories (a) Continue in existing form (b) Continue with some Modifications (suggest modifications) (c) scale up the scheme (Financial/Physical/ both) (d) Scale down the scheme (Financial/Physical/ both) (e) Close (f) Merge with another scheme as sub-scheme/ component. [ for further details see comments for para 5. 1] RRR RRREVALUATION OF SCHEMES - CONCEPTUAL NOTE Evaluation contributes fundamentally to sound public governance. It can help governments improve the design and implementation of public policies that, in turn, can lead to prosperity for their country and well-being for citizens. Evaluation contributes to promoting public accountability, learning and increased public sector effectiveness through improved decision-making. Evaluation is critical to ensure that policies are actually improving outcomes by bringing an understanding of what works, why, for whom, and under what circumstances. 2. Budget Speech of 2016 spelt out the vision of the Government with regard to the appraisal and approval cycle that CSSs and CSs would need to follow after the end of the planning cycle, of which 12% Five Year Plan was the last. Government mandated that ‘every new scheme being sanctioned by Government will have a sunset date and outcome review’.! For the XVth FC cycle NITI Aayog had undertaken evaluation of 28 umbrella schemes. Evaluation of other CSs/CSSs schemes was undertaken by the respective Ministries/Departments. For the current FC cycle too, NITI Aayog is conducting evaluation of CSSs. Evaluation Criteria ce Broadly, evaluation of schemes can be categorised into six — relevance, coherence, effectiveness, efficiency, impact and sustainability. 2 It is important that the definitions of the criteria are understood within the broader context to make evaluation useful and of high quality. There are two basic guiding principles which must be applied to any evaluation- a. that it should be contextualized ( what is that the evaluation is trying to find out and what is the intended purpose of evaluation?) b. time and resources may be deployed based on the purpose of evaluation which may take into account data availability, timing and methodological. considerations. Relevance - Relevance is a pertinent consideration across the programme or policy cycle from design to implementation. It can be analysed via four potential elements for analysis: relevance to beneficiary and stakeholder needs, relevance to context, relevance of quality and design, and relevance over time. They should be included as required for the purpose of the evaluation and are not exhaustive. * Budget Speech 2016 (Para 110) . https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm Page 1 of 4 Gs)The evaluation of relevance should start by determining whether the objectives of the intervention are adequately defined, realistic and feasible, and whether the results are verifiable and aligned with current international standards for development interventions. Coherence — Coherence brings in increased focus on the synergies (or trade-offs) between policy areas and the growing attention to cross-government co- ordination. It identifies situations where duplication of efforts or inconsistencies in approaches to implementing policies across government or different institutions can undermine overall progress. This criterion also encourages evaluators to understand the role of an intervention within a particular system (organisation, sector, thematic area, country), as opposed to taking an exclusively intervention- or institution-centric perspective. Effectiveness - Effectiveness helps in understanding the extent to which an intervention is achieving or has achieved its objectives. It can provide insight into whether an intervention has attained its planned results (OOMFP), the process by which this was done, which factors were decisive in this process and whether there were any unintended effects. Efficiency - This criterion is an opportunity to check whether an intervention’s resources can be justified by its results, which is of major practical and political importance. Efficiency is about choices between feasible alternatives that can deliver similar results within the given resources. Before cost-effectiveness comparisons can be made, alternatives must be identified that are genuinely feasible and comparable in terms of quality and results. Impact - Under this criteria ToRs must be designed so as to allow for’such information to be captures as to whether or not the intervention created change that really matters to people. It is an opportunity to take a broader perspective and a holistic view. Questions that the impact criterion might cover include: . Has the intervention caused a significant change in the lives of the intended beneficiaries? . How did the intervention cause higher-level effects (such as changes in norms or systems)? . Did all the intended target groups, including the most disadvantaged and vulnerable, benefit equally from the intervention? . Is the intervention transformative - does it create enduring changes in norms — including gender norms - and systems, whether intended or not? Page 2of4. Is the intervention leading to other changes, including “scalable” or “replicable” results? . How will the intervention contribute to changing society for the better? Sustainability - The role of evaluation here can be to scrutinise assumptions in the theory of change for how sustainability is achieved. One important question which needs to be asked is whether the impact will be sustainable once fiscal support from the government is removed. Evaluation of Schemes - DoE’s specific requirements 4. Evaluation of public interventions and policies needs to recognize the scarcity of resources and fiscal affordability. An evaluation which concludes that more financial ‘support for a scheme would make it effective in achieving its outputs and outcomes, misses the core point of limitation of resources and the zero sum game’ that plays out in resource allocation, In a developing country like ours which faces multiple social, economic and environmental challenges, availability of public resources amongst competing demands will always be limited. Priority setting and resource allocation are vital questions to be addressed, and evaluation needs to assist in this. While interventions may be chosen to maximize specific objectives, they all are constrained with respect to practical and budgetary issues. Hence, evaluation recommendations should, at least as a base case, assume constant real level of budget and should not assume enhanced funding. If felt essential, alternative non-base case recommendations may be made separately. Ss The evaluation exercise should, inter alia, aim at reducing/ rationalising excessive number of Centrally Sponsored Schemes and other schemes (also recommended by Expenditure Management Commission and Finance Commission), avoiding waste arising from duplication and ‘opening of shops’ by each Ministry, reduction in wasteful expenses (including excessive use of consultants, administrative overheads etc.). Managing of funds that flow from the Centre to the States for implementation of CSSs and implementing agencies under CSs also needs to be factored to avoid parking of funds by agencies/ states. While the template ToR attempts to capture the essence of all the above listed concepts, DoE recommends that ToRs for evaluation of schemes may be modified to adequately reflect these issues enabling the evaluation process to capture such data/information. 6. Hence, apart from fiscal affordability, evaluation of schemes should focus on: Page 3 of 4Improving value for money through identification of avoidable duplication and overlaps between different schemes HL. Assessing the continued relevance and their effectiveness and efficiency in achieving the intended objectives iil. Reduction in the number of schemes, especially schemes of small size, through closure/ merger of irrelevant/ low value-adding schemes iv. Reduction in avoidable overhead expenditure on consultants, programme management, administration etc. te Based on the data/information captured during the evaluation process the following must be identified and appropriate action taken: e Schemes with similar objectives/overlaps (intra / inter Ministry /Department) must be considered for merger/closure * Schemes having backward and forward linkages ( schemes, which are presently being implemented independently, but will be more effective if they are combined/merged) must be considered for merger. * Schemes which have lost relevance over the period of its implementation (this may include a component of the scheme as well) must be considered for closure. Schemes with minimal prima facie impact on account of inadequate budget must be considered for closure. * Schemes with low budgetary utilization must be considered for closure. Page 4o0f4Ge HIX Riel PRASEEP K. SINHA FRA SRGR CABINET SECRETARY GOVERNMENT OF INDIA D.O No. 1/50/1/2016-Cab 19” December, 2017 b> » iste e eD - Dear Secretary, aa" Ww is et al ar i This is regarding proposals submitted by various Ministries/ Departments refating to oc fo n tt hi en u 1a 4ti "o n F io nf a C ne cn et r Ca oll my m iS sp so in oso nr (e id .e . S fc rh oe mm 2e 0s 1 7b -e 1y 8o n tod 2t 0h 1e 9 1 -2 20” ) .P l Tan h eP e fr oi lo lod w a inn gd pt oil it nt th se nt aer vm e lp P, ay emerged ‘during their consideration: Ay) aoe (i Continuation of Schemes/ Programmes should not be undertaken as a IRE routine exercise and any extension beyond 2016-17 should be proposed only Wee after appropriate evaluation. (i) A restructuring of the Scheme/ Programme needs to be considered or the basis of evaluation report and giving it a mew nomenclature accordingly may also be explored. Citi} In Schemes/ Prograrimes where unspent balances at the State level or at the implementing agency tevel have come to notice, the extension of the term and the justification for enhancement of annual allocation would néed to be properly scrutinized. ‘ 2: It has also been observed that many Ministries/ Departments have already released budgetary funds during the year 2017-18 without following the prescribed procedure and obtaining authorization of expenditure beyond the 12” Plan period. This may not only create an anomalous situation but may also present a fait accompli for extension of such Schemes/ Programmes upto the year 2019-20. A In this context, | would like to re-emphasize that proposats of this nature should be preceded by an appropriate evaluation of the Scheme/ Programme and necessary - restructuring carried out in them so as to achieve the desired results. It may be noted that non-compliance with the above would delay the processing of such proposals for approval by Cabinet) CCEA. 4. You are, therefore, requested to ensure necessary compliance in this regard. With regards, ’ Yours sincerely, Shri Ajay Narayan Jha (P. K Sinha) Secretary, Department of Expenditure New Delhi htrapati Bhavan, New Delt \ 6 -23018633 E-mail cabinetsy(@nic inAppendix-I List of CS being evaluated by NITI Aayog S.No. | Name of the | Name of the Scheme Ministry/Department 1. | M/o Parliamentary Affairs National e-Vidhan (NeVA) 2. |M/o Health and Family | Ayushman Bharat - HWC (AAM) Welfare 3. M/o Agriculture and Farmers | PM Kisan Welfare 4. | D/o Financial Services PM Stand up India 5. | D/o Financial Services PM Jan Dhan Yojana 6. D/o Financial Services PM Jeevan Jyoti Yojana (PMJJY) 7. | D/o Financial Services PM Suraksha Bima Yojana (PMSBY) 8. | Department for Promotion of | Refund of CGST and IGST to Industry and Internal Trade | Industrial Units in NE region and (DPIIT) Himalayan States 9. |M/o Petroleum and Natural | PM Ujjwala Yojana Gas (MoPNG)