A (Statistical) Evaluation of the Max Weber Programme
By Igor Guardiancich, Post-doctoral Fellow, Collegio Carlo Alberto
The academic year 2011-12 marked the 6th anniversary of the Max Weber Post-doctoral Programme (MWP) of the European University Institute (EUI), the largest post-doctoral programme in the social sciences in Europe. Due to its size and increasing popularity – circa 40 Fellows are being admitted each year, out of more than 1,000 applicants – Ramon Marimon (MWP Director), Alexey Bessudnov (2010-11 MW Fellow) and I (Research Assistant at the Academic Careers Observatory in 2009-11) decided that it was high time we evaluated the effectiveness of the MWP to provide Fellows with the tools to successfully enter the labour market.
The passage between obtaining a PhD and being firmly anchored to a (often academic) post is widely recognized as a complex endeavor. Recent PhD graduates face a high degree of uncertainty during the first couple of years and are frequently employed in post-doctoral and other temporary positions (increasingly in the social sciences and also in the humanities). Moreover, anecdotal evidence shows that the global financial crisis has only exacerbated the problems associated with an overproduction of PhDs and, sometimes, with the mismatch between the skills of graduates and what the market requires.
As the MWP is a structured programme aimed at improving the academic and research skills of its participants, it is particularly fitting to check whether it does, in fact, facilitate this transition. A question of method arises at this point. Against whom should the past cohorts of MW Fellows be compared with in order to meaningfully evaluate their success? Not entering into excessive detail, we employed a method called propensity score matching, which (as the name implies) matches the Fellows with their closest counterparts, based on a number of core characteristics. As the most important trait for the selection of an applicant is her ‘quality’, based on publications, the research proposal, and so on (in addition to career interests as well the availability of mentorship at the EUI), it becomes natural to compare the Fellows with those applicants to the MWP (the control group), who have similar characteristics and were, hence, close to admission but – for a number of reasons, mostly capacity – where in the end not selected.
So, this first selection was based on all the application packages received in 2007-10. It was followed by two surveys sent to 117 past Fellows and 261 former applicants (close contenders), asking them detailed questions on their career advancement, current placement, publication record and their satisfaction with their job, research, teaching and, more generally, their lives. The number of responses we received was satisfactory: 79 Fellows and 91 applicants answered the whole survey. At this point we were ready to compare the outcomes of the two groups.
The comparison yielded three significant results, to varying degrees. The Fellows reported on average higher levels of overall life satisfaction, they proved to have a relatively better publication record than the control group, and their satisfaction with teaching was greater. As publishing and teaching are two fields where the Fellows were given particularly robust training, the results corroborated the Programme’s overall quality.
As for overall satisfaction, even though we cannot unambiguously state that the MWP is the deus ex machina that makes the Fellows’ lives better, we also reject the simple explanation that they are satisfied because they spent a year in Florence, which is almost universally appreciated for its good climate and culturally stimulating environment.
Our preferred explanation is the long-term effect of the MWP on the academic adjustment of the Fellows. At the EUI, they are given training in many aspects of academic life, including teaching, academic writing, publication strategies, time management, and so on. Fellows from different disciplines, universities and countries were given the opportunity to communicate and share their academic experiences. We argue that this affected their long-term psychological and professional adjustment to academic life, helped them to deal with the stress of the early stages of their academic careers, and positively affected their general life satisfaction. We are not aware of any similar rigorous assessment of a graduate or post-doctoral programme following its first five years of existence.