《国家职业教育改革实施方案》（简称“职教20条”）确定在职业院校和应用型本科高校启动“学历证书+职业技能等级证书”（即“1+X”证书）试点工作，这是我国证书制度建设方面的一次重要探索。“1+X”证书制度建设是否成功，在很大程度上取决于职业技能评价的质量，而评价方法对技能评价结果的真实性与可信度具有直接影响。近年来，国内外在职业技能评价方面进行了很多研究和开发工作，形成了多种职业技能评价方法，为职业技能的评价和发展提供了重要基础，但是评价方法和工具的质量始终存在问题，无法保证评价的科学性和可信度。本研究聚焦X职业技能评价方法的质量，系统探讨以下问题：（1）现有职业技能评价方法的质量如何？（2）什么是有效的X职业技能评价方法？（3）如何对X职业技能评价方法的有效性开展元评价？（4）元评价的结果是什么？（5）什么原因造成目前这一结果？（6）未来应该如何改进X职业技能评价方法？

研究以设计导向职业教育思想为指导，在信度理论、整体效度观、建构主义和情境学习理论的基础上，借鉴语言测试使用论证框架，初步建构X职业技能评价方法的有效性论证框架，并采用德尔菲法确定有效性论证框架的指标。研究按照X职业技能等级证书所属职业类别的不同，选取技术类的J证书和D证书，服务类的S证书、G证书和Z证书（其中S证书是以人为服务对象、G、Z证书是以技术为服务对象）作为元评价对象。根据已构建的有效性论证框架，设计元评价证据列表、元评价等级评判表、访谈提纲和观察表。采用文本分析法、访谈法、观察法和专家评判法，收集五个证书评价方法有效性的相关证据资料。将证据整理归类，分别填入五个证书的证据列表中，采用图尔敏论证模式，对证据列表中证据的充分程度进行论证和等级评判，根据等级赋值，计算出证据（三级指标）对理由（二级指标）和主张（一级指标）的支持程度分值，从而获得有效性论证框架中各级指标的达成度情况，画出指标达成度雷达图，并进行分析与讨论。

元评价结果显示：（1）五个证书评价方法的整体有效性差异不大；（2）没有足够证据表明五个证书的评价方法在“测试分数有效性”方面达成度高，说明证书设计者对评价方法信度的考量不够；（3）五个证书在“测试分数解释有效性”方面达成度不同，说明不同证书效度不同；（4）没有足够证据表明五个证书在“测试结果使用有效性”方面达成度高，但多数证书在保证学习机会公平性方面做得较好；（5）没有足够证据表明五个证书在“测试及其结果使用影响的有效性”方面达成度高，但测试对考生和职业院校具有一定有益影响。

The "National Reform Implementation Plan of Vocational Education " (referred to as "20 Vocational Education") has determined to launch the pilot work of "Education Certificate + Professional Competence Level Certificate" (ie "1+X" certificate) in vocational schools and applied undergraduate colleges. It is an important exploration in the construction of our country's certificate system. To a large extent, the success of the "1+X" certification system depends on the quality of the professional competence evaluation, and the evaluation method has a direct impact on the authenticity and credibility of the competence evaluation results. In recent years, a lot of research and development work has been carried out on professional competence evaluation at home and abroad, and a variety of professional competence evaluation methods have been formed, which provide an important foundation for the evaluation and development of professional competence. However, the quality of evaluation methods and tools has always been problematic. The scientificity and credibility of the evaluation cannot be guaranteed. This research focuses on the quality of X professional competence evaluation methods, and systematically explores the following questions: (1) What is the quality of the existing professional competence evaluation methods? (2) What is the validity of X professional competence evaluation method? (3) How to carry out meta-evaluation of the validity of X professional competence evaluation method? (4) What is the result of meta-evaluation? (5) What caused the current result? (6) How to improve the validity of X professional competence evaluation methods in the future?

The research is guided by the idea of design-oriented vocational education, on the basis of reliability theory, unified validity, constructivism and contextual learning theory, and borrows the “Assessment Use Argument” in language testing to initially construct the validity argumentation framework of the X professional competence evaluation method. We used the Delphi method to determine the validity argumentation framework indicators.

According to the different occupational categories of the X professional competence level certificate, the research selects the J certificate and D certificate of the technical category, the S certificate, the G certificate and the Z certificate of the service category (where the S certificate is for people as the service object, and the G and Z certificates are for Technology as the service object) as the meta-evaluation object. According to the established validity argumentation framework, design meta-evaluation evidence list, meta-evaluation grade evaluation table, interview outline and observation table. Using text analysis method, interview method, observation method and expert evaluation method, collect the relevant evidence data of the validity of the five certificate evaluation methods. We sorted and categorized the evidence and filled them into the evidence lists of the five certificates. Using Toulmin’s argumentation model, we demonstrated and graded the adequacy of the evidence in the evidence list. According to the grade, we calculated the evidence (three-level indicator) Scores of support for reasons (second-level indicators) and claims (first-level indicators), so as to obtain the degree of achievement of all indicators in the validity argumentation framework, draw a radar chart of the degree of achievement of the indicators, and conduct analysis and discussion.

The meta-evaluation results show that: (1) There is no significant difference in the overall validity of different certificate evaluation methods; (2) There is not enough evidence to show that the evaluation methods of the five certificates have a high degree of achievement in the "test score validity", indicating the design of the certificate insufficient consideration of the reliability of the evaluation method; (3) Different certificates have different degrees of achievement in the "explanatory validity of test scores", indicating that the validity of the five certificates is different; (4) There is insufficient evidence to show that the five certificates have a high degree of achievement in the "validity of results", but most certificates do a good job in ensuring the fairness of learning opportunities; (5) There is not enough evidence to show that the five certificates have a high degree of achievement in the "validity of the test and the use of results", but the test has a certain beneficial impact on candidates and vocational colleges.

Analyzing and discussing the results of meta-evaluation, we draw the following main conclusions: (1) The five certificates do a good job in the fairness of the organization; (2) When designing standards, the evaluation organization contacts the enterprise and focuses on practical experience; (3) The test has a certain impact on candidates and vocational colleges, but it has limited impact on promoting teaching reform and improving students' professional competence; (4) There are certain differences in professional competence evaluation methods for different vocational categories; (5) The design of professional competence standards is lacking theoretical basis and the guidance of scientific method; (6) The design of X professional competence evaluation method still lacks support of reliability theory and validity theory; (7) Pure result-oriented evaluation cannot fully evaluate professional cognitive competence; (8) Simple computer-aided testing cannot realize the evaluation of professional competence.

In order to improve the validity of X professional competence evaluation methods, we should: (1) take reliability and validity theories as the theoretical basis for the design of evaluation methods; (2) pay attention to theoretical basis and science method in the development of professional competence standards; (3) adopt situational test task and bring in authenticity evaluation; (4) combine result evaluation with process evaluation, use multiple evaluation methods; (5) use artificial intelligence and other technologies to optimize computer-aided testing; (6) use different competence evaluation methods between different professional categories Certificates.