1
00:00:00,860 --> 00:00:05,700
uh welcome to the uh inauspicious start

2
00:00:03,600 --> 00:00:07,440
to the conference on reproducibility and

3
00:00:05,700 --> 00:00:09,059
replicability and economics and social

4
00:00:07,440 --> 00:00:11,580
sciences

5
00:00:09,059 --> 00:00:14,219
um our goal was with a conference is to

6
00:00:11,580 --> 00:00:15,660
provide a consistent series of

7
00:00:14,219 --> 00:00:17,279
discussions by Specialists and

8
00:00:15,660 --> 00:00:19,199
practitioners on topics of

9
00:00:17,279 --> 00:00:21,420
reproducibility replicability of

10
00:00:19,199 --> 00:00:23,039
transparency and their intersection with

11
00:00:21,420 --> 00:00:24,779
a variety of topics

12
00:00:23,039 --> 00:00:27,240
over the course of the next year we'll

13
00:00:24,779 --> 00:00:28,680
have panels discussing educational and

14
00:00:27,240 --> 00:00:31,619
procedural barriers slowing down

15
00:00:28,680 --> 00:00:34,020
adoption of reproducible methods whether

16
00:00:31,619 --> 00:00:35,340
journals or institutions or funders

17
00:00:34,020 --> 00:00:37,920
should be the verifiers of

18
00:00:35,340 --> 00:00:39,600
reproducibility whether and how

19
00:00:37,920 --> 00:00:41,520
scientists work can be made to be

20
00:00:39,600 --> 00:00:43,200
reproducible at every stage of the

21
00:00:41,520 --> 00:00:45,540
research process including at the

22
00:00:43,200 --> 00:00:48,360
Inception and data collection stage

23
00:00:45,540 --> 00:00:50,219
and implications of all of the above for

24
00:00:48,360 --> 00:00:52,559
the training of undergraduates and

25
00:00:50,219 --> 00:00:54,300
graduate students and in many cases of

26
00:00:52,559 --> 00:00:56,160
continuing education for researchers in

27
00:00:54,300 --> 00:00:58,559
the field

28
00:00:56,160 --> 00:01:00,480
um I'd like to introduce Alex Machuda

29
00:00:58,559 --> 00:01:02,399
who you still see on on screen who's my

30
00:01:00,480 --> 00:01:04,680
co-pi on this and host of some of the

31
00:01:02,399 --> 00:01:07,619
future webinars

32
00:01:04,680 --> 00:01:09,299
um Marie Canali is uh off screen is one

33
00:01:07,619 --> 00:01:10,560
of my co-organizers Ian schwannee who

34
00:01:09,299 --> 00:01:12,299
can make it today is another one of

35
00:01:10,560 --> 00:01:14,700
those

36
00:01:12,299 --> 00:01:17,100
um and finally Sarah Brooks who is off

37
00:01:14,700 --> 00:01:20,460
screen who keeps the wheels rolling um

38
00:01:17,100 --> 00:01:22,320
Alex's Fair will be Manning the QA and

39
00:01:20,460 --> 00:01:26,400
relay any questions that you type in

40
00:01:22,320 --> 00:01:28,560
there to us panelists so please ask

41
00:01:26,400 --> 00:01:30,840
questions all throughout the webinar uh

42
00:01:28,560 --> 00:01:33,680
we'll have uh explicitly time at the end

43
00:01:30,840 --> 00:01:36,600
to address uh all those questions

44
00:01:33,680 --> 00:01:38,820
uh so today's webinar uh we're starting

45
00:01:36,600 --> 00:01:40,259
off with is uh on the topic on the

46
00:01:38,820 --> 00:01:42,780
general topic of institutional support

47
00:01:40,259 --> 00:01:44,400
with a specific question should journals

48
00:01:42,780 --> 00:01:46,500
verify reproducibility

49
00:01:44,400 --> 00:01:48,060
uh so we're looking forward to hearing

50
00:01:46,500 --> 00:01:49,740
from our expert panelists into your

51
00:01:48,060 --> 00:01:52,140
questions

52
00:01:49,740 --> 00:01:54,420
um we'll have about 40 minutes of

53
00:01:52,140 --> 00:01:56,820
discussion uh by and amongst the

54
00:01:54,420 --> 00:01:59,460
panelists and then we'll turn to

55
00:01:56,820 --> 00:02:01,799
audience questions so again to submit a

56
00:01:59,460 --> 00:02:03,299
question please type it into the QA and

57
00:02:01,799 --> 00:02:06,299
we will ask your question the question

58
00:02:03,299 --> 00:02:08,459
on your behalf uh to the speakers

59
00:02:06,299 --> 00:02:09,539
without further Ado let me introduce our

60
00:02:08,459 --> 00:02:12,660
panelists

61
00:02:09,539 --> 00:02:15,720
uh today we're joined

62
00:02:12,660 --> 00:02:16,860
um by three panelists

63
00:02:15,720 --> 00:02:19,800
um

64
00:02:16,860 --> 00:02:21,660
first off uh with embence uh professor

65
00:02:19,800 --> 00:02:23,520
of Economics at the school of humanities

66
00:02:21,660 --> 00:02:25,140
and Sciences senior fellow at the

67
00:02:23,520 --> 00:02:27,599
Stanford Institute for economic policy

68
00:02:25,140 --> 00:02:29,580
research Coulter family faculty fellow

69
00:02:27,599 --> 00:02:31,739
at Stanford University and relevant for

70
00:02:29,580 --> 00:02:34,459
this particular discussion editor of

71
00:02:31,739 --> 00:02:34,459
econometrica

72
00:02:34,500 --> 00:02:37,920
um also joining us is Tim Salman

73
00:02:36,599 --> 00:02:39,840
professor of Economics at Southern

74
00:02:37,920 --> 00:02:42,300
Methodist University I am the editor of

75
00:02:39,840 --> 00:02:43,620
economic inquiry which is the one of the

76
00:02:42,300 --> 00:02:44,760
journals of the western economic

77
00:02:43,620 --> 00:02:47,700
Association

78
00:02:44,760 --> 00:02:48,959
and finally uh Tony Whitehead uh Dale

79
00:02:47,700 --> 00:02:50,879
that I came up professor of business

80
00:02:48,959 --> 00:02:52,140
business administration at the Ross

81
00:02:50,879 --> 00:02:54,180
School of Business at the University of

82
00:02:52,140 --> 00:02:56,280
Michigan and relevant for this

83
00:02:54,180 --> 00:02:57,840
discussion editor of the Journal of

84
00:02:56,280 --> 00:03:01,620
financial economics

85
00:02:57,840 --> 00:03:02,700
finally my name is larsville Huber

86
00:03:01,620 --> 00:03:04,200
um I'm

87
00:03:02,700 --> 00:03:06,599
um executive director of the labor

88
00:03:04,200 --> 00:03:08,700
Dynamics Institute at Cornell's ilr

89
00:03:06,599 --> 00:03:10,620
school and for this discussion relevant

90
00:03:08,700 --> 00:03:12,300
data editor of the eight journals of the

91
00:03:10,620 --> 00:03:14,099
American economic Association I'll be

92
00:03:12,300 --> 00:03:16,800
the moderator so I will let our

93
00:03:14,099 --> 00:03:20,459
panelists talk and with that I'll turn

94
00:03:16,800 --> 00:03:25,080
it over to Guido who will start

95
00:03:20,459 --> 00:03:28,739
thanks uh last uh thanks very much for

96
00:03:25,080 --> 00:03:30,540
having me on this uh this panel

97
00:03:28,739 --> 00:03:33,000
um to be honest when I started as

98
00:03:30,540 --> 00:03:35,159
editors three years ago replication

99
00:03:33,000 --> 00:03:39,540
policies were not something I had really

100
00:03:35,159 --> 00:03:41,400
given uh a lot of thoughts uh to

101
00:03:39,540 --> 00:03:43,620
have in the last couple of years I've

102
00:03:41,400 --> 00:03:45,120
been convinced that both that this is a

103
00:03:43,620 --> 00:03:47,760
very important issue for the profession

104
00:03:45,120 --> 00:03:49,920
as a whole but also the journal the

105
00:03:47,760 --> 00:03:52,620
journals kind of really should make a

106
00:03:49,920 --> 00:03:54,060
difference uh here and the AAA of course

107
00:03:52,620 --> 00:03:57,200
has shown a great deal of leadership

108
00:03:54,060 --> 00:04:01,200
here in a econometric society we've been

109
00:03:57,200 --> 00:04:04,260
happy to free ride so far

110
00:04:01,200 --> 00:04:06,840
but now and we will continue to kind of

111
00:04:04,260 --> 00:04:09,540
draw on the experience of the the aea

112
00:04:06,840 --> 00:04:12,480
data out of the team that Lars has been

113
00:04:09,540 --> 00:04:14,580
leading but we'll be more proactive in

114
00:04:12,480 --> 00:04:18,000
this area and we're going to also insist

115
00:04:14,580 --> 00:04:21,120
on replicating all papers in economic

116
00:04:18,000 --> 00:04:24,540
and the other economic Society journals

117
00:04:21,120 --> 00:04:27,060
prior to publication and we will soon

118
00:04:24,540 --> 00:04:28,080
be hiring a data Editor to take charge

119
00:04:27,060 --> 00:04:30,660
of that

120
00:04:28,080 --> 00:04:32,759
and while it'll take us a while to to

121
00:04:30,660 --> 00:04:35,360
get to the point that we do a scooter

122
00:04:32,759 --> 00:04:39,120
job as the AAA journals

123
00:04:35,360 --> 00:04:41,520
then by the end of uh of the three years

124
00:04:39,120 --> 00:04:43,680
that I have left in my term as editor I

125
00:04:41,520 --> 00:04:46,320
hope and expect that we'll we will be

126
00:04:43,680 --> 00:04:48,840
reproducing all empirical papers uh that

127
00:04:46,320 --> 00:04:51,479
we published at least once uh without

128
00:04:48,840 --> 00:04:53,780
confidential data exactly how we deal

129
00:04:51,479 --> 00:04:56,040
with confidential data

130
00:04:53,780 --> 00:04:58,020
it's not completely clear to me at the

131
00:04:56,040 --> 00:05:00,840
moment but we'll probably try to follow

132
00:04:58,020 --> 00:05:03,120
what the AAA is doing there in that

133
00:05:00,840 --> 00:05:05,280
respect but at least for the publicly

134
00:05:03,120 --> 00:05:08,400
available they for papers with publicly

135
00:05:05,280 --> 00:05:11,639
available data we will make sure that

136
00:05:08,400 --> 00:05:13,919
the numbers do reproduce prior to uh

137
00:05:11,639 --> 00:05:15,180
full acceptance of a paper prior to

138
00:05:13,919 --> 00:05:18,660
publication

139
00:05:15,180 --> 00:05:20,880
and so the one reason I think this is

140
00:05:18,660 --> 00:05:23,940
really important is that um

141
00:05:20,880 --> 00:05:26,160
I see the role of journals these days

142
00:05:23,940 --> 00:05:28,320
not so much as disseminating the papers

143
00:05:26,160 --> 00:05:30,120
that can easily be done by the authors

144
00:05:28,320 --> 00:05:32,699
themselves by posting on their own

145
00:05:30,120 --> 00:05:35,940
websites or by posting on archive type

146
00:05:32,699 --> 00:05:38,759
websites is that what what the journals

147
00:05:35,940 --> 00:05:40,800
do is really give the paper some stamp

148
00:05:38,759 --> 00:05:43,680
of approval and we do that by having

149
00:05:40,800 --> 00:05:45,180
reviewers read the papers and evaluate

150
00:05:43,680 --> 00:05:47,100
them

151
00:05:45,180 --> 00:05:49,020
but kind of traditionally the reviewers

152
00:05:47,100 --> 00:05:51,120
Focus only on the theoretical arguments

153
00:05:49,020 --> 00:05:53,580
and the and the reported results they

154
00:05:51,120 --> 00:05:55,199
may ask authors to do an additional

155
00:05:53,580 --> 00:05:57,240
analysis

156
00:05:55,199 --> 00:05:58,919
but they don't really check those and

157
00:05:57,240 --> 00:06:00,180
when we publish papers we essentially

158
00:05:58,919 --> 00:06:03,180
say that we think the theoretical

159
00:06:00,180 --> 00:06:05,160
results are both interesting and and

160
00:06:03,180 --> 00:06:07,259
correct and that still may not always

161
00:06:05,160 --> 00:06:09,360
pan out and we sometimes need to publish

162
00:06:07,259 --> 00:06:12,240
corrections but at least there and kind

163
00:06:09,360 --> 00:06:14,820
of there's there's a clear hope and a

164
00:06:12,240 --> 00:06:17,160
plan that that the published theoretical

165
00:06:14,820 --> 00:06:19,380
results are correct but traditionally we

166
00:06:17,160 --> 00:06:21,300
don't really question the validity of

167
00:06:19,380 --> 00:06:22,020
the numbers in a paper in the paper at

168
00:06:21,300 --> 00:06:24,539
all

169
00:06:22,020 --> 00:06:26,220
and given that now the technology is

170
00:06:24,539 --> 00:06:28,139
there to actually do that

171
00:06:26,220 --> 00:06:29,639
there's no reason to also say a

172
00:06:28,139 --> 00:06:31,440
publication that we think the empirical

173
00:06:29,639 --> 00:06:33,300
results in the paper

174
00:06:31,440 --> 00:06:36,060
are correct at least kind of in The

175
00:06:33,300 --> 00:06:39,360
Limited sense that they they follow from

176
00:06:36,060 --> 00:06:41,699
the data and the codes that the the

177
00:06:39,360 --> 00:06:44,460
authors have produced and so I think we

178
00:06:41,699 --> 00:06:46,919
should do that we ought to do that

179
00:06:44,460 --> 00:06:49,560
and not doing that is is really their

180
00:06:46,919 --> 00:06:52,740
election of our of our duty given that

181
00:06:49,560 --> 00:06:54,660
it's actually possible to do that now

182
00:06:52,740 --> 00:06:56,699
over time I would actually like to go

183
00:06:54,660 --> 00:06:59,460
further and give the reviewers the

184
00:06:56,699 --> 00:07:00,960
opportunity to actually get access to

185
00:06:59,460 --> 00:07:03,539
the data of all the papers and the

186
00:07:00,960 --> 00:07:05,940
review but that that may be a longer

187
00:07:03,539 --> 00:07:08,160
term thing

188
00:07:05,940 --> 00:07:09,840
one additional comment I want to make uh

189
00:07:08,160 --> 00:07:12,720
here is that

190
00:07:09,840 --> 00:07:15,180
as more and more journals are following

191
00:07:12,720 --> 00:07:17,819
the the lead of the aea in this regard

192
00:07:15,180 --> 00:07:19,620
we really need to update how we train

193
00:07:17,819 --> 00:07:21,419
graduate students so that they fully

194
00:07:19,620 --> 00:07:24,180
understand from the beginning of their

195
00:07:21,419 --> 00:07:26,940
graduate student careers that any data

196
00:07:24,180 --> 00:07:28,319
analysis they do and any data analysis

197
00:07:26,940 --> 00:07:30,300
that they hope to eventually publish

198
00:07:28,319 --> 00:07:33,120
needs to be reproducible and

199
00:07:30,300 --> 00:07:35,940
documentable documented

200
00:07:33,120 --> 00:07:38,940
and really in a shape that other people

201
00:07:35,940 --> 00:07:41,220
can use that and and build on those uh

202
00:07:38,940 --> 00:07:42,660
on that code that's not how I was

203
00:07:41,220 --> 00:07:44,280
trained when I was in graduate school

204
00:07:42,660 --> 00:07:45,539
that's not how I was trying this I'm The

205
00:07:44,280 --> 00:07:48,479
Graduate

206
00:07:45,539 --> 00:07:51,660
but in the future we will need to do

207
00:07:48,479 --> 00:07:53,039
that and do that in a way that will

208
00:07:51,660 --> 00:07:55,440
improve and streamline the

209
00:07:53,039 --> 00:07:57,599
reproducibility process and in general

210
00:07:55,440 --> 00:07:59,400
improve the the quality of the the

211
00:07:57,599 --> 00:08:01,860
empirical work

212
00:07:59,400 --> 00:08:04,080
so I think that we really need to kind

213
00:08:01,860 --> 00:08:06,180
of enroll the graduate programs make

214
00:08:04,080 --> 00:08:08,340
sure that the students understand what

215
00:08:06,180 --> 00:08:10,919
is going to be expected of them in the

216
00:08:08,340 --> 00:08:14,039
in the long run whichever way they're

217
00:08:10,919 --> 00:08:18,360
going to be doing uh empirical work

218
00:08:14,039 --> 00:08:22,340
so let me uh stop here and and uh and

219
00:08:18,360 --> 00:08:22,340
let someone else make some comments

220
00:08:24,660 --> 00:08:27,599
okay

221
00:08:25,979 --> 00:08:30,539
um thank you Guido

222
00:08:27,599 --> 00:08:33,539
so next up is uh Tim Tim I think you had

223
00:08:30,539 --> 00:08:35,880
a a small number of slides to share

224
00:08:33,539 --> 00:08:36,959
um why don't you start with that

225
00:08:35,880 --> 00:08:39,180
okay

226
00:08:36,959 --> 00:08:41,520
all right uh well thank you for inviting

227
00:08:39,180 --> 00:08:43,740
me onto the panel I think this is a very

228
00:08:41,520 --> 00:08:45,660
important topic coincidentally I was

229
00:08:43,740 --> 00:08:47,640
just lecturing to my undergraduates

230
00:08:45,660 --> 00:08:48,839
about it today so it's uh fresh on my

231
00:08:47,640 --> 00:08:49,500
mind

232
00:08:48,839 --> 00:08:52,560
um

233
00:08:49,500 --> 00:08:53,580
so I took over as editor of economic

234
00:08:52,560 --> 00:08:56,820
inquiry

235
00:08:53,580 --> 00:08:58,320
uh year year and a half ago and this

236
00:08:56,820 --> 00:09:01,140
issue about

237
00:08:58,320 --> 00:09:04,560
um data requirements and reproducibility

238
00:09:01,140 --> 00:09:06,120
was on my mind and um it's one of the I

239
00:09:04,560 --> 00:09:08,880
introduced a lot of new policies this

240
00:09:06,120 --> 00:09:09,660
was one of the first ones so

241
00:09:08,880 --> 00:09:11,279
um

242
00:09:09,660 --> 00:09:12,959
what I thought I would talk about is

243
00:09:11,279 --> 00:09:15,180
that you know yes I think the journal

244
00:09:12,959 --> 00:09:16,800
should verify reproducibility

245
00:09:15,180 --> 00:09:19,440
um let me go quickly through some of the

246
00:09:16,800 --> 00:09:22,500
arguments I heard against it give some

247
00:09:19,440 --> 00:09:24,120
of my uh comments on how I dealt with

248
00:09:22,500 --> 00:09:27,000
those arguments and well then talk about

249
00:09:24,120 --> 00:09:28,980
some of the uh other reasons why I think

250
00:09:27,000 --> 00:09:31,800
that it's important to do it

251
00:09:28,980 --> 00:09:33,360
um so this I mean it is not still not

252
00:09:31,800 --> 00:09:35,519
exactly something that everyone agrees

253
00:09:33,360 --> 00:09:37,920
on so it's um worth at least

254
00:09:35,519 --> 00:09:39,660
understanding some of those issues so

255
00:09:37,920 --> 00:09:42,180
one of the most common

256
00:09:39,660 --> 00:09:43,380
um objections I got when uh I was

257
00:09:42,180 --> 00:09:45,779
talking to people about whether we

258
00:09:43,380 --> 00:09:48,360
should do it is authors being concerned

259
00:09:45,779 --> 00:09:51,779
well if you make me post my code in my

260
00:09:48,360 --> 00:09:52,440
data other people can copy my work

261
00:09:51,779 --> 00:09:55,019
um

262
00:09:52,440 --> 00:09:56,279
my point to that of yes that's exactly

263
00:09:55,019 --> 00:09:58,019
the point

264
00:09:56,279 --> 00:10:00,000
um once your paper is published you want

265
00:09:58,019 --> 00:10:02,160
people to be able to understand how you

266
00:10:00,000 --> 00:10:03,860
did it uh why you did it and to be able

267
00:10:02,160 --> 00:10:06,600
to um

268
00:10:03,860 --> 00:10:09,300
start from there and go forward it makes

269
00:10:06,600 --> 00:10:11,160
you more work more impactful

270
00:10:09,300 --> 00:10:13,860
um and instead of being a negative I

271
00:10:11,160 --> 00:10:15,420
think this is pretty much the idea

272
00:10:13,860 --> 00:10:18,060
um it's uh it's useful to be able to do

273
00:10:15,420 --> 00:10:20,100
it so it allows people to build off of

274
00:10:18,060 --> 00:10:22,440
it which uh I think is what we're trying

275
00:10:20,100 --> 00:10:24,920
to do as scientists is try to build some

276
00:10:22,440 --> 00:10:27,839
tools build some science that others can

277
00:10:24,920 --> 00:10:29,339
build off of so I don't see that as a

278
00:10:27,839 --> 00:10:30,500
negative

279
00:10:29,339 --> 00:10:34,019
um

280
00:10:30,500 --> 00:10:35,399
uh the other common issue I heard that

281
00:10:34,019 --> 00:10:37,800
people were concerned about is the

282
00:10:35,399 --> 00:10:40,380
degree to which this might place a extra

283
00:10:37,800 --> 00:10:42,420
burden on authors and

284
00:10:40,380 --> 00:10:43,920
um that uh from the journal side that

285
00:10:42,420 --> 00:10:45,420
this might decrease submissions as

286
00:10:43,920 --> 00:10:48,660
people looking at the requirements might

287
00:10:45,420 --> 00:10:50,700
choose to uh submit elsewhere

288
00:10:48,660 --> 00:10:52,140
um and this is definitely an issue uh in

289
00:10:50,700 --> 00:10:55,260
particular for

290
00:10:52,140 --> 00:10:57,120
um you know for the AAA journals and for

291
00:10:55,260 --> 00:10:59,579
econometrica

292
00:10:57,120 --> 00:11:01,260
they can require authors to submit on

293
00:10:59,579 --> 00:11:03,779
blue polka dot paper with the paper

294
00:11:01,260 --> 00:11:06,779
carried in by carrier pigeons and people

295
00:11:03,779 --> 00:11:09,060
will still submit for journals that are

296
00:11:06,779 --> 00:11:11,480
in the set of not econometrica and not

297
00:11:09,060 --> 00:11:11,480
AER

298
00:11:11,820 --> 00:11:17,579
it's a legitimate concern that authors

299
00:11:14,339 --> 00:11:19,800
might uh um not want to satisfy uh very

300
00:11:17,579 --> 00:11:21,000
Draconian requirements

301
00:11:19,800 --> 00:11:21,680
so

302
00:11:21,000 --> 00:11:25,019
um

303
00:11:21,680 --> 00:11:26,880
you know an important point there uh to

304
00:11:25,019 --> 00:11:29,640
uh Echo something that Guido was talking

305
00:11:26,880 --> 00:11:33,360
about if authors learn to do this from

306
00:11:29,640 --> 00:11:34,860
the beginning which they should then

307
00:11:33,360 --> 00:11:37,860
um complying with these requirements is

308
00:11:34,860 --> 00:11:39,720
not a big deal and if authors begin the

309
00:11:37,860 --> 00:11:41,700
project to be reproducible my view is

310
00:11:39,720 --> 00:11:44,160
it's useful for the authors as well I

311
00:11:41,700 --> 00:11:45,839
mean you know if you uh get a revised

312
00:11:44,160 --> 00:11:47,700
and resubmit on a paper a year and a

313
00:11:45,839 --> 00:11:49,920
half after you've done the work for it

314
00:11:47,700 --> 00:11:52,620
it's useful for yourself to have the

315
00:11:49,920 --> 00:11:54,480
scripts to go back and look through so I

316
00:11:52,620 --> 00:11:55,980
I think it's useful for the authors to

317
00:11:54,480 --> 00:11:56,940
begin with

318
00:11:55,980 --> 00:11:59,519
um

319
00:11:56,940 --> 00:12:01,500
and uh like I said it's more of an issue

320
00:11:59,519 --> 00:12:02,700
for some journals than others and I

321
00:12:01,500 --> 00:12:04,880
think this is one of the reasons why

322
00:12:02,700 --> 00:12:06,899
some journals might make different

323
00:12:04,880 --> 00:12:09,420
decisions than others we can talk more

324
00:12:06,899 --> 00:12:10,200
about that later

325
00:12:09,420 --> 00:12:13,380
um

326
00:12:10,200 --> 00:12:15,540
and also another point about uh what

327
00:12:13,380 --> 00:12:18,060
aguir was saying some people argue that

328
00:12:15,540 --> 00:12:20,220
well you can satisfy this requirement

329
00:12:18,060 --> 00:12:21,959
that any authors who want to make this

330
00:12:20,220 --> 00:12:23,339
stuff transparent should be able to do

331
00:12:21,959 --> 00:12:24,839
so they can put the data on their own

332
00:12:23,339 --> 00:12:26,880
website they can host their own

333
00:12:24,839 --> 00:12:29,040
repositories

334
00:12:26,880 --> 00:12:31,920
which they can but it doesn't accomplish

335
00:12:29,040 --> 00:12:34,019
the goals of what we want with this

336
00:12:31,920 --> 00:12:36,420
um so for the goals

337
00:12:34,019 --> 00:12:38,339
in my view it's necessary to allow for

338
00:12:36,420 --> 00:12:40,800
the more rapid and reliable scientific

339
00:12:38,339 --> 00:12:42,540
advancement right once someone figures

340
00:12:40,800 --> 00:12:44,100
out how to do something it's useful for

341
00:12:42,540 --> 00:12:48,120
others to be able to build off of that

342
00:12:44,100 --> 00:12:49,980
and progress with it uh having someone

343
00:12:48,120 --> 00:12:51,540
else spend a year and a half Reinventing

344
00:12:49,980 --> 00:12:53,760
the wheel of what someone else did is

345
00:12:51,540 --> 00:12:55,380
not always that reliable right the point

346
00:12:53,760 --> 00:12:57,540
of publishing the papers is to put the

347
00:12:55,380 --> 00:13:00,959
work out there so that others can build

348
00:12:57,540 --> 00:13:02,639
off of it and having journals verifying

349
00:13:00,959 --> 00:13:05,220
that these things work

350
00:13:02,639 --> 00:13:07,139
and verifying the methods I think help

351
00:13:05,220 --> 00:13:08,940
with with that

352
00:13:07,139 --> 00:13:12,000
um and the other reason that I really

353
00:13:08,940 --> 00:13:14,459
think that journals have to uh host

354
00:13:12,000 --> 00:13:16,620
these things themselves is the

355
00:13:14,459 --> 00:13:18,480
importance of this in determining

356
00:13:16,620 --> 00:13:20,820
intentional and unintentional fraud

357
00:13:18,480 --> 00:13:22,920
right I mean we all know about many

358
00:13:20,820 --> 00:13:25,320
cases of intentional fraud of people

359
00:13:22,920 --> 00:13:27,660
fabricating data sets or going on

360
00:13:25,320 --> 00:13:29,639
gigantic uh P hacking and fishing

361
00:13:27,660 --> 00:13:32,160
Expeditions for the foundation of their

362
00:13:29,639 --> 00:13:34,440
paper this is certainly not something

363
00:13:32,160 --> 00:13:35,339
that anyone wants in any reputable

364
00:13:34,440 --> 00:13:37,560
Journal

365
00:13:35,339 --> 00:13:41,639
having these requirements at the journal

366
00:13:37,560 --> 00:13:43,200
I think would uh not only allow you to

367
00:13:41,639 --> 00:13:45,420
identify some of this if it's happening

368
00:13:43,200 --> 00:13:46,500
but deter people from doing it to begin

369
00:13:45,420 --> 00:13:49,920
with

370
00:13:46,500 --> 00:13:51,420
um and so I think it's vital for uh the

371
00:13:49,920 --> 00:13:53,700
Integrity of our research to make

372
00:13:51,420 --> 00:13:55,740
certain the journals do this I think it

373
00:13:53,700 --> 00:13:57,959
also helps deter what I would call

374
00:13:55,740 --> 00:14:00,720
unintentional or accidental P hacking

375
00:13:57,959 --> 00:14:02,639
right which is people essentially doing

376
00:14:00,720 --> 00:14:04,800
a lot of different regressions and then

377
00:14:02,639 --> 00:14:07,920
choosing the one that happens to have

378
00:14:04,800 --> 00:14:08,639
the lowest p-value

379
00:14:07,920 --> 00:14:11,160
um

380
00:14:08,639 --> 00:14:13,440
based upon some of the arguments that

381
00:14:11,160 --> 00:14:16,740
it's slightly better than others this is

382
00:14:13,440 --> 00:14:18,839
a long conversation as well but I think

383
00:14:16,740 --> 00:14:21,420
I think having the journals have the

384
00:14:18,839 --> 00:14:22,620
ability to verify the data I think it uh

385
00:14:21,420 --> 00:14:26,100
helps

386
00:14:22,620 --> 00:14:27,720
um to uh convince authors that both of

387
00:14:26,100 --> 00:14:29,160
these types of bad behavior shouldn't be

388
00:14:27,720 --> 00:14:29,880
done

389
00:14:29,160 --> 00:14:32,579
um

390
00:14:29,880 --> 00:14:35,880
related to this one of the main reasons

391
00:14:32,579 --> 00:14:37,380
why I wanted to do it is this point who

392
00:14:35,880 --> 00:14:41,040
wants to be the only Journal not

393
00:14:37,380 --> 00:14:43,260
requiring uh the the data to be provided

394
00:14:41,040 --> 00:14:45,899
and to be verified

395
00:14:43,260 --> 00:14:48,180
um if you are one of the remaining

396
00:14:45,899 --> 00:14:49,860
journals uh not requiring this it leads

397
00:14:48,180 --> 00:14:51,660
to selection dynamics that I don't think

398
00:14:49,860 --> 00:14:53,940
I have to talk about too much but they

399
00:14:51,660 --> 00:14:56,100
are not in the favor of the journal

400
00:14:53,940 --> 00:14:58,860
um so as more and more journals start to

401
00:14:56,100 --> 00:15:01,139
do it it decreases the deterrent um of

402
00:14:58,860 --> 00:15:03,000
people submitting because people should

403
00:15:01,139 --> 00:15:05,100
start accepting that they're going to

404
00:15:03,000 --> 00:15:08,519
have to deal with these issues anywhere

405
00:15:05,100 --> 00:15:10,920
and uh the few journals that uh remain

406
00:15:08,519 --> 00:15:13,380
not doing this well

407
00:15:10,920 --> 00:15:16,740
those fraudsters which journals are they

408
00:15:13,380 --> 00:15:20,120
going to choose so um this is definitely

409
00:15:16,740 --> 00:15:20,120
something to uh

410
00:15:20,160 --> 00:15:23,639
um think about the last point I'll make

411
00:15:22,320 --> 00:15:25,500
is that

412
00:15:23,639 --> 00:15:27,360
um it also is important to allow for

413
00:15:25,500 --> 00:15:30,180
verification of what generated results

414
00:15:27,360 --> 00:15:32,519
so I'll mention uh one paper that we

415
00:15:30,180 --> 00:15:35,220
published at EI last year

416
00:15:32,519 --> 00:15:37,339
um in which they um had multiple

417
00:15:35,220 --> 00:15:39,959
different uh research teams try to

418
00:15:37,339 --> 00:15:41,699
replicate an existing paper by just

419
00:15:39,959 --> 00:15:44,579
giving them the data giving them the

420
00:15:41,699 --> 00:15:47,279
question and say go for it right now in

421
00:15:44,579 --> 00:15:49,560
engaging in any empirical work there are

422
00:15:47,279 --> 00:15:51,720
a ton of uh decisions you have to make

423
00:15:49,560 --> 00:15:53,940
about what to do with the data how to

424
00:15:51,720 --> 00:15:56,339
deal with what appears to be mistakes in

425
00:15:53,940 --> 00:15:57,899
the data how to deal with outliers how

426
00:15:56,339 --> 00:15:59,579
to deal with things

427
00:15:57,899 --> 00:16:01,740
and if it's not clear of what you've

428
00:15:59,579 --> 00:16:03,600
done with that it's difficult for other

429
00:16:01,740 --> 00:16:05,339
people to know what you did and why they

430
00:16:03,600 --> 00:16:06,839
might have gotten inconsistent results

431
00:16:05,339 --> 00:16:09,120
from yours

432
00:16:06,839 --> 00:16:11,000
um so this paper showed that different

433
00:16:09,120 --> 00:16:13,800
research teams came up with results

434
00:16:11,000 --> 00:16:14,480
positive negative with

435
00:16:13,800 --> 00:16:17,339
um

436
00:16:14,480 --> 00:16:19,079
standard errors all over the board even

437
00:16:17,339 --> 00:16:20,699
using the original data and research

438
00:16:19,079 --> 00:16:23,160
question of the researchers because

439
00:16:20,699 --> 00:16:24,480
people made different choices about uh

440
00:16:23,160 --> 00:16:26,760
what data what they did with different

441
00:16:24,480 --> 00:16:29,220
data points having a record of that is

442
00:16:26,760 --> 00:16:31,079
important for other people to do this

443
00:16:29,220 --> 00:16:33,740
research there was another paper that

444
00:16:31,079 --> 00:16:36,899
came through EI a while ago

445
00:16:33,740 --> 00:16:39,120
where some people were trying to start

446
00:16:36,899 --> 00:16:40,459
off with someone else's estimation and

447
00:16:39,120 --> 00:16:43,740
build from there

448
00:16:40,459 --> 00:16:45,480
it turned out that the original paper

449
00:16:43,740 --> 00:16:47,820
had basically gotten a positive effect

450
00:16:45,480 --> 00:16:49,380
this replication could only come up with

451
00:16:47,820 --> 00:16:50,699
a negative effect and then they tried to

452
00:16:49,380 --> 00:16:53,339
build off of it

453
00:16:50,699 --> 00:16:56,100
and even though one of the uh original

454
00:16:53,339 --> 00:16:58,500
authors was a referee on this paper even

455
00:16:56,100 --> 00:17:00,120
they couldn't figure out why

456
00:16:58,500 --> 00:17:01,560
um the other people couldn't come up

457
00:17:00,120 --> 00:17:04,500
with the same result

458
00:17:01,560 --> 00:17:08,400
but if the original authors had posted

459
00:17:04,500 --> 00:17:09,900
their scripts posted their data

460
00:17:08,400 --> 00:17:12,000
um you know it would have allowed for

461
00:17:09,900 --> 00:17:13,260
those following up to make certain that

462
00:17:12,000 --> 00:17:15,540
they're starting from the same point

463
00:17:13,260 --> 00:17:18,299
okay and in my view in the long run

464
00:17:15,540 --> 00:17:20,880
these types of things are important for

465
00:17:18,299 --> 00:17:22,559
the Integrity of our work and for

466
00:17:20,880 --> 00:17:25,020
um scientific advancements so I will

467
00:17:22,559 --> 00:17:26,240
stop there

468
00:17:25,020 --> 00:17:28,140
um

469
00:17:26,240 --> 00:17:30,419
and uh

470
00:17:28,140 --> 00:17:34,200
turn it over to uh

471
00:17:30,419 --> 00:17:36,720
Tony good Tony thank you Tim and uh with

472
00:17:34,200 --> 00:17:38,880
that our last panelist Tony

473
00:17:36,720 --> 00:17:40,679
you've got the floor

474
00:17:38,880 --> 00:17:42,840
and I do

475
00:17:40,679 --> 00:17:44,160
hang on one sec

476
00:17:42,840 --> 00:17:46,799
there we go

477
00:17:44,160 --> 00:17:49,440
so I wanted to start with a little bit

478
00:17:46,799 --> 00:17:52,020
of a discussion about how this

479
00:17:49,440 --> 00:17:54,360
um uh issue started up in finance we're

480
00:17:52,020 --> 00:17:57,240
a little bit behind economics

481
00:17:54,360 --> 00:17:59,940
um it was first proposed in 2011 and all

482
00:17:57,240 --> 00:18:02,700
three journals summarily rejected this

483
00:17:59,940 --> 00:18:05,160
idea of code and data sharing

484
00:18:02,700 --> 00:18:07,919
so that was at the end of it in 2015 I

485
00:18:05,160 --> 00:18:11,340
was a part of a three Journal group that

486
00:18:07,919 --> 00:18:13,380
was tasked with a proposal of coming up

487
00:18:11,340 --> 00:18:15,900
with a code and data sharing

488
00:18:13,380 --> 00:18:17,100
um protocol for the Jaffe the JF and the

489
00:18:15,900 --> 00:18:19,799
RFS

490
00:18:17,100 --> 00:18:22,500
we ended up hunting on data sharing

491
00:18:19,799 --> 00:18:25,799
because a lot of Finance data is both

492
00:18:22,500 --> 00:18:28,140
propriety proprietary and very expensive

493
00:18:25,799 --> 00:18:30,720
but we did come up with a code sharing

494
00:18:28,140 --> 00:18:34,559
policy and the American Finance

495
00:18:30,720 --> 00:18:37,260
Association approved a version in 2016.

496
00:18:34,559 --> 00:18:39,539
and the jfe editorial board dismissed

497
00:18:37,260 --> 00:18:41,220
the idea and so the SFS Council and it

498
00:18:39,539 --> 00:18:42,840
was interesting to see some of the

499
00:18:41,220 --> 00:18:45,780
objectives

500
00:18:42,840 --> 00:18:48,120
it these are some quotes this imposes

501
00:18:45,780 --> 00:18:50,340
costs on the researcher without any

502
00:18:48,120 --> 00:18:52,080
obvious benefits

503
00:18:50,340 --> 00:18:53,760
while encouraging greater quality

504
00:18:52,080 --> 00:18:56,340
control is a worthy goal I think there

505
00:18:53,760 --> 00:18:58,200
are already abundant incentives from the

506
00:18:56,340 --> 00:19:00,480
risk of losing one's reputation within

507
00:18:58,200 --> 00:19:03,000
the peer-reviewed community

508
00:19:00,480 --> 00:19:04,140
this is Draconian are there documented

509
00:19:03,000 --> 00:19:06,660
problems

510
00:19:04,140 --> 00:19:08,940
I can't imagine anybody me making sense

511
00:19:06,660 --> 00:19:10,320
of the code now it is documented for

512
00:19:08,940 --> 00:19:12,299
people who have been working on it but

513
00:19:10,320 --> 00:19:14,880
someone who comes up with comes out of

514
00:19:12,299 --> 00:19:18,179
fresh requires lots of guidance so there

515
00:19:14,880 --> 00:19:21,000
was an enormous amount of

516
00:19:18,179 --> 00:19:23,160
objections I don't particularly find

517
00:19:21,000 --> 00:19:24,500
these arguments

518
00:19:23,160 --> 00:19:27,419
um

519
00:19:24,500 --> 00:19:29,340
satisfactory and for many of the reasons

520
00:19:27,419 --> 00:19:31,260
that Tim said

521
00:19:29,340 --> 00:19:33,600
but this isn't the end of the door story

522
00:19:31,260 --> 00:19:35,940
The RFS adopted the Journal of finance

523
00:19:33,600 --> 00:19:37,980
code policy in 2020

524
00:19:35,940 --> 00:19:41,580
at that point I was also on a different

525
00:19:37,980 --> 00:19:43,820
AFA task force to discuss to propose

526
00:19:41,580 --> 00:19:46,380
extending the Journal of Finance

527
00:19:43,820 --> 00:19:47,940
co-chairing policy to data and we came

528
00:19:46,380 --> 00:19:49,919
up with the

529
00:19:47,940 --> 00:19:51,960
um data infrastructure

530
00:19:49,919 --> 00:19:53,700
proposal and then the pandemic happened

531
00:19:51,960 --> 00:19:55,919
and then

532
00:19:53,700 --> 00:19:59,280
I don't know what happened but the RFS

533
00:19:55,919 --> 00:20:01,039
adopted the code policy in 2020 and I

534
00:19:59,280 --> 00:20:04,559
took over the jfe

535
00:20:01,039 --> 00:20:07,620
in July 2021 and that's the first thing

536
00:20:04,559 --> 00:20:09,240
I did is I adopted I went one step

537
00:20:07,620 --> 00:20:12,539
further than the other two journals I

538
00:20:09,240 --> 00:20:14,520
adopted a code and data policy and so

539
00:20:12,539 --> 00:20:18,179
Lars asked me why it's because I've

540
00:20:14,520 --> 00:20:21,840
wanted to do this for oh 20 like 11

541
00:20:18,179 --> 00:20:25,620
years and it just I finally got to do it

542
00:20:21,840 --> 00:20:27,780
so we think reproducibility is important

543
00:20:25,620 --> 00:20:30,380
unfortunately we don't have enough

544
00:20:27,780 --> 00:20:34,080
resources for in-house reproduction I've

545
00:20:30,380 --> 00:20:36,299
looked at how much this costs and we

546
00:20:34,080 --> 00:20:39,020
can't do it given our budget

547
00:20:36,299 --> 00:20:41,580
so we decided that code and

548
00:20:39,020 --> 00:20:43,799
non-proprietary data disclosure was a

549
00:20:41,580 --> 00:20:45,660
close substitute

550
00:20:43,799 --> 00:20:48,419
so but I think this

551
00:20:45,660 --> 00:20:50,220
discussion of the history in finance

552
00:20:48,419 --> 00:20:51,539
brings up the costs and benefits of

553
00:20:50,220 --> 00:20:55,200
reproducibility

554
00:20:51,539 --> 00:20:57,360
so as the other two panelists have

555
00:20:55,200 --> 00:20:59,160
emphasized I think it enhances the

556
00:20:57,360 --> 00:21:01,380
credibility of Science and it really

557
00:20:59,160 --> 00:21:03,179
benefits students students who can go

558
00:21:01,380 --> 00:21:06,840
look at code and data that have been

559
00:21:03,179 --> 00:21:09,419
disclosed as are it's enormously useful

560
00:21:06,840 --> 00:21:11,760
for them I actually think also it

561
00:21:09,419 --> 00:21:14,880
benefits individual researchers by

562
00:21:11,760 --> 00:21:16,559
enforcing good research procedures I

563
00:21:14,880 --> 00:21:19,980
certainly conduct research different

564
00:21:16,559 --> 00:21:22,380
than I did 30 years ago I use GitHub I

565
00:21:19,980 --> 00:21:24,720
use Version Control I'm super careful

566
00:21:22,380 --> 00:21:26,580
and I think that's good I think we

567
00:21:24,720 --> 00:21:28,440
should teach our graduate students to do

568
00:21:26,580 --> 00:21:31,440
this and I think all of us old farts

569
00:21:28,440 --> 00:21:33,960
should change our ways

570
00:21:31,440 --> 00:21:36,720
um I think it does solve the problem of

571
00:21:33,960 --> 00:21:38,100
potential for unseen errors and I think

572
00:21:36,720 --> 00:21:41,400
most importantly it improves

573
00:21:38,100 --> 00:21:43,200
transparency you can write a paper very

574
00:21:41,400 --> 00:21:45,480
clearly and

575
00:21:43,200 --> 00:21:48,360
there will be nobody who will be able to

576
00:21:45,480 --> 00:21:50,460
completely replicate your paper because

577
00:21:48,360 --> 00:21:52,200
there's a lot of decisions in the code

578
00:21:50,460 --> 00:21:55,200
but if you replicate the code that

579
00:21:52,200 --> 00:21:57,480
becomes in some sense an auxiliary text

580
00:21:55,200 --> 00:22:00,360
and so I think it's great

581
00:21:57,480 --> 00:22:02,760
these benefits don't come out with costs

582
00:22:00,360 --> 00:22:06,840
it does consume resources

583
00:22:02,760 --> 00:22:08,820
there's an there is a um

584
00:22:06,840 --> 00:22:12,600
argument out there that I don't find

585
00:22:08,820 --> 00:22:14,340
entirely unpersuasive which is that the

586
00:22:12,600 --> 00:22:17,520
important papers get replicated anyway

587
00:22:14,340 --> 00:22:19,559
I'm going to come back to that

588
00:22:17,520 --> 00:22:21,600
an argument that I don't particularly

589
00:22:19,559 --> 00:22:23,820
find persuasive is that if there's code

590
00:22:21,600 --> 00:22:25,980
and data availability people will go on

591
00:22:23,820 --> 00:22:29,539
fishing Expeditions and try to harass

592
00:22:25,980 --> 00:22:29,539
people to give them gotchas

593
00:22:30,480 --> 00:22:34,080
but all of this brings up something

594
00:22:32,400 --> 00:22:36,299
though the way I think of this is we

595
00:22:34,080 --> 00:22:37,500
have a public good and it's costly to

596
00:22:36,299 --> 00:22:40,320
produce

597
00:22:37,500 --> 00:22:43,380
and so should Journal who should pay

598
00:22:40,320 --> 00:22:45,360
this the pregubian subsidy it we have a

599
00:22:43,380 --> 00:22:46,799
good it's a it's a good thing should it

600
00:22:45,360 --> 00:22:48,659
be the journals but there are other

601
00:22:46,799 --> 00:22:51,059
Alternatives should it be universities

602
00:22:48,659 --> 00:22:54,120
should it be granting

603
00:22:51,059 --> 00:22:56,940
um agencies and then how should big

604
00:22:54,120 --> 00:22:58,980
should this subsidy be should it just be

605
00:22:56,940 --> 00:23:01,740
code and data sharing and we should

606
00:22:58,980 --> 00:23:03,240
leave the reproduction to the profession

607
00:23:01,740 --> 00:23:06,960
at Large

608
00:23:03,240 --> 00:23:08,640
should it be actual Journal reproduction

609
00:23:06,960 --> 00:23:10,980
or should it be data and code

610
00:23:08,640 --> 00:23:12,480
verification I'll come back to that in a

611
00:23:10,980 --> 00:23:14,880
bit because I want to explain exactly

612
00:23:12,480 --> 00:23:17,220
what I mean by that later

613
00:23:14,880 --> 00:23:19,860
and then should the subsidy to apply to

614
00:23:17,220 --> 00:23:21,840
all published papers

615
00:23:19,860 --> 00:23:24,480
so let me talk about that who should pay

616
00:23:21,840 --> 00:23:26,580
it is expensive I've costed this out my

617
00:23:24,480 --> 00:23:29,100
original intention when I started when I

618
00:23:26,580 --> 00:23:31,380
took over the jfe was to have

619
00:23:29,100 --> 00:23:33,480
um somebody like Lars on hand and I

620
00:23:31,380 --> 00:23:36,419
decided that that was way too costly and

621
00:23:33,480 --> 00:23:38,460
that we couldn't do it so that was the

622
00:23:36,419 --> 00:23:41,100
end of that and I think that is probably

623
00:23:38,460 --> 00:23:44,100
true for most journals

624
00:23:41,100 --> 00:23:45,600
what about granting agencies this is

625
00:23:44,100 --> 00:23:47,460
something that would certainly work in

626
00:23:45,600 --> 00:23:49,200
the physical sciences or the biological

627
00:23:47,460 --> 00:23:52,020
sciences because they all run off of

628
00:23:49,200 --> 00:23:55,320
Grants but in economics and certainly in

629
00:23:52,020 --> 00:23:57,960
finance very few people get grants

630
00:23:55,320 --> 00:24:00,179
what about universities I think that's a

631
00:23:57,960 --> 00:24:02,039
real possibility because I think you

632
00:24:00,179 --> 00:24:06,000
could change the culture that this would

633
00:24:02,039 --> 00:24:08,220
be part of research it's not just

634
00:24:06,000 --> 00:24:10,919
producing the paper it's producing the

635
00:24:08,220 --> 00:24:13,620
paper plus something that could be could

636
00:24:10,919 --> 00:24:16,080
be reproduced and so perhaps in the

637
00:24:13,620 --> 00:24:19,280
future universities could be persuaded

638
00:24:16,080 --> 00:24:19,280
to take part in this

639
00:24:19,380 --> 00:24:24,539
and then what should the subsidy be

640
00:24:22,440 --> 00:24:26,340
code couldn't data verification

641
00:24:24,539 --> 00:24:28,200
reproduction

642
00:24:26,340 --> 00:24:29,940
I think until there's a culture of

643
00:24:28,200 --> 00:24:32,760
reproducibility

644
00:24:29,940 --> 00:24:35,400
I am in actually I I if there were

645
00:24:32,760 --> 00:24:36,900
infinite resources I would say that all

646
00:24:35,400 --> 00:24:39,360
journals should

647
00:24:36,900 --> 00:24:41,460
produce results I think that would be a

648
00:24:39,360 --> 00:24:44,400
great idea but I don't think that that

649
00:24:41,460 --> 00:24:46,740
is universally available

650
00:24:44,400 --> 00:24:48,600
I also don't think that just code and

651
00:24:46,740 --> 00:24:50,940
data disclosure are enough because

652
00:24:48,600 --> 00:24:52,140
they're often ill documented they're

653
00:24:50,940 --> 00:24:55,580
missing

654
00:24:52,140 --> 00:24:59,039
I have checked the the

655
00:24:55,580 --> 00:25:01,020
nine submissions of code and data that

656
00:24:59,039 --> 00:25:02,460
the jiv have gotten so far because we

657
00:25:01,020 --> 00:25:06,260
grandfathered in all of the old

658
00:25:02,460 --> 00:25:06,260
submissions but I can't do that forever

659
00:25:06,960 --> 00:25:10,559
and then as I said reproduction is

660
00:25:09,419 --> 00:25:12,539
costly

661
00:25:10,559 --> 00:25:14,539
it's also and I think this is important

662
00:25:12,539 --> 00:25:17,400
it's very

663
00:25:14,539 --> 00:25:19,440
costly for researchers without abundant

664
00:25:17,400 --> 00:25:21,780
Ras I believe I've had an argument on

665
00:25:19,440 --> 00:25:23,640
Twitter with Lars about this

666
00:25:21,780 --> 00:25:26,640
um I certainly don't have abundant Ras

667
00:25:23,640 --> 00:25:29,640
and I'm in a pretty good University and

668
00:25:26,640 --> 00:25:33,360
I think that this does impose costs on

669
00:25:29,640 --> 00:25:36,720
researchers that it magnifies the um

670
00:25:33,360 --> 00:25:38,279
these funding differences between really

671
00:25:36,720 --> 00:25:40,320
really good universities and less

672
00:25:38,279 --> 00:25:42,360
well-funded universities and it's not

673
00:25:40,320 --> 00:25:43,860
obvious to me that we want to magnify

674
00:25:42,360 --> 00:25:46,260
that

675
00:25:43,860 --> 00:25:48,419
so I am in favor of code and data

676
00:25:46,260 --> 00:25:50,940
verification is there a readme file this

677
00:25:48,419 --> 00:25:53,820
is what I check is there an execution

678
00:25:50,940 --> 00:25:56,340
script is there data or pseudo data does

679
00:25:53,820 --> 00:25:58,679
it look like it could be used and then

680
00:25:56,340 --> 00:26:00,600
perhaps the for many journals that can't

681
00:25:58,679 --> 00:26:04,020
afford this the rest of the profession

682
00:26:00,600 --> 00:26:05,820
could be the reproducers

683
00:26:04,020 --> 00:26:07,200
but I want to talk about one thing other

684
00:26:05,820 --> 00:26:09,059
thing that I think is more important

685
00:26:07,200 --> 00:26:12,000
which is secret data which is a big

686
00:26:09,059 --> 00:26:14,700
problem in finance and should journals

687
00:26:12,000 --> 00:26:16,740
use discourage the use of secret data

688
00:26:14,700 --> 00:26:19,260
I'm not talking about administrative

689
00:26:16,740 --> 00:26:21,419
data like the census or commercial data

690
00:26:19,260 --> 00:26:24,000
like compustat it's not secret it's just

691
00:26:21,419 --> 00:26:25,919
costly I'm talking about data that your

692
00:26:24,000 --> 00:26:28,440
friend at the bank gave you

693
00:26:25,919 --> 00:26:29,940
and I think that's a different question

694
00:26:28,440 --> 00:26:32,820
that is more important for

695
00:26:29,940 --> 00:26:34,980
reproducibility is should this use of

696
00:26:32,820 --> 00:26:37,260
secret data come with some sort of

697
00:26:34,980 --> 00:26:40,440
reproducibility requirement should we

698
00:26:37,260 --> 00:26:42,960
require authors that use secret data to

699
00:26:40,440 --> 00:26:45,240
have it be accessible on say an air

700
00:26:42,960 --> 00:26:48,059
gapped computer if somebody wanted to

701
00:26:45,240 --> 00:26:50,640
try to reproduce it and so I think that

702
00:26:48,059 --> 00:26:53,480
that is one of the bigger issues

703
00:26:50,640 --> 00:26:53,480
and I'm done

704
00:26:54,600 --> 00:26:59,700
thank you Tony

705
00:26:56,880 --> 00:27:02,580
um and so thank you to all panelists for

706
00:26:59,700 --> 00:27:06,000
a really uh interesting and covering a

707
00:27:02,580 --> 00:27:08,159
white swath here of what we're doing

708
00:27:06,000 --> 00:27:10,860
um yes I do remember our our discussion

709
00:27:08,159 --> 00:27:12,539
on Twitter about the resources necessary

710
00:27:10,860 --> 00:27:14,640
um and let me take the moderator's

711
00:27:12,539 --> 00:27:16,760
prerogative here

712
00:27:14,640 --> 00:27:16,760
um

713
00:27:16,919 --> 00:27:21,179
I kind of see the aspect of it's

714
00:27:19,380 --> 00:27:22,740
difficult to reproduce as part of that

715
00:27:21,179 --> 00:27:25,320
equilibrium we have them get all

716
00:27:22,740 --> 00:27:27,659
upgraded our skill set and to what

717
00:27:25,320 --> 00:27:29,220
extent can technology substitute for

718
00:27:27,659 --> 00:27:30,779
some of that for instance if you're

719
00:27:29,220 --> 00:27:33,600
submitting something

720
00:27:30,779 --> 00:27:35,400
can we require that essentially we have

721
00:27:33,600 --> 00:27:36,840
a machine check that there's push button

722
00:27:35,400 --> 00:27:39,179
reproduce built to which then becomes

723
00:27:36,840 --> 00:27:41,700
trivial to do at some level

724
00:27:39,179 --> 00:27:43,440
I'm going to right now put forward that

725
00:27:41,700 --> 00:27:46,260
can't possibly solve all the possible

726
00:27:43,440 --> 00:27:47,820
solutions and we don't have uniform ways

727
00:27:46,260 --> 00:27:50,940
of doing that but I think it's something

728
00:27:47,820 --> 00:27:54,600
to put forward as is that something we

729
00:27:50,940 --> 00:27:57,419
want to slowly move to or not because it

730
00:27:54,600 --> 00:27:59,940
would reduce the cost of reproducing a

731
00:27:57,419 --> 00:28:02,159
reproducibility checking at the at the

732
00:27:59,940 --> 00:28:03,840
journal level but it might also not make

733
00:28:02,159 --> 00:28:06,179
it necessary because if it's also easy

734
00:28:03,840 --> 00:28:07,860
to do it the journals with the

735
00:28:06,179 --> 00:28:09,360
universities will do it the researchers

736
00:28:07,860 --> 00:28:11,220
will do it Etc

737
00:28:09,360 --> 00:28:13,520
that might be a desirable equilibrium

738
00:28:11,220 --> 00:28:15,900
not easy to move to

739
00:28:13,520 --> 00:28:17,640
comments from the panelists yeah last

740
00:28:15,900 --> 00:28:19,980
can I actually I want to make some

741
00:28:17,640 --> 00:28:22,620
comments on some of the issues that uh

742
00:28:19,980 --> 00:28:25,620
Tim and uh and Tony raised

743
00:28:22,620 --> 00:28:27,480
I mean the Tim kind of mentioned that

744
00:28:25,620 --> 00:28:29,700
sort of the cost to the authors of

745
00:28:27,480 --> 00:28:32,159
actually uh doing this and Tony

746
00:28:29,700 --> 00:28:33,120
mentioned this as well but so at some

747
00:28:32,159 --> 00:28:36,900
level

748
00:28:33,120 --> 00:28:39,480
I'm not really very sympathetic to that

749
00:28:36,900 --> 00:28:42,000
argument kind of on the on the author's

750
00:28:39,480 --> 00:28:44,159
path it feels to me a little bit like

751
00:28:42,000 --> 00:28:45,059
supposedly we're in an equilibrium where

752
00:28:44,159 --> 00:28:47,279
people

753
00:28:45,059 --> 00:28:49,260
wrote down theorems but they didn't

754
00:28:47,279 --> 00:28:51,299
actually include the proofs and they

755
00:28:49,260 --> 00:28:53,340
said you know what you need to trust me

756
00:28:51,299 --> 00:28:55,919
it would be a lot of work to write the

757
00:28:53,340 --> 00:28:57,779
the proof and so you're making it more

758
00:28:55,919 --> 00:28:59,460
costly for me to write these papers and

759
00:28:57,779 --> 00:29:02,460
I'm going to send it somewhere else

760
00:28:59,460 --> 00:29:04,140
we don't do that we we say no you do

761
00:29:02,460 --> 00:29:08,220
need you make a statement you do need to

762
00:29:04,140 --> 00:29:10,980
have a proof and so the same way you put

763
00:29:08,220 --> 00:29:12,720
in results in a paper we want to see to

764
00:29:10,980 --> 00:29:15,000
the extent possible that these results

765
00:29:12,720 --> 00:29:17,539
actually are are meaningful

766
00:29:15,000 --> 00:29:20,340
and now

767
00:29:17,539 --> 00:29:22,740
that that's sort of at some you know

768
00:29:20,340 --> 00:29:24,480
checking that does come with a there's a

769
00:29:22,740 --> 00:29:27,419
substantial cost and for the the

770
00:29:24,480 --> 00:29:29,460
economic society that was a big concern

771
00:29:27,419 --> 00:29:31,200
as well as we actually had a discussion

772
00:29:29,460 --> 00:29:33,980
at some point but uh

773
00:29:31,200 --> 00:29:37,140
we should charge the office for for

774
00:29:33,980 --> 00:29:39,179
replicating their papers and you know

775
00:29:37,140 --> 00:29:42,179
the people didn't want to necessarily

776
00:29:39,179 --> 00:29:43,919
impose those costs but we we discussed

777
00:29:42,179 --> 00:29:45,179
one version of that when we would make

778
00:29:43,919 --> 00:29:48,059
it voluntary

779
00:29:45,179 --> 00:29:51,299
we would allow the authors to choose

780
00:29:48,059 --> 00:29:54,899
between paying a fee to get the paper

781
00:29:51,299 --> 00:29:56,640
replicated uh or not but then the paper

782
00:29:54,899 --> 00:29:58,620
would be published kind of with a gold

783
00:29:56,640 --> 00:30:01,140
star saying this one actually

784
00:29:58,620 --> 00:30:02,159
was replicable and this one we don't

785
00:30:01,140 --> 00:30:03,899
really know what the numbers mean

786
00:30:02,159 --> 00:30:05,880
because we never we never could check it

787
00:30:03,899 --> 00:30:08,460
it's sort of clear that if you if you

788
00:30:05,880 --> 00:30:10,740
were to do that and we we didn't go that

789
00:30:08,460 --> 00:30:12,720
in that direction and I'm not sure I

790
00:30:10,740 --> 00:30:13,980
would be in favor of that but if you

791
00:30:12,720 --> 00:30:16,159
went in that direction it's sort of

792
00:30:13,980 --> 00:30:17,820
clear that the authors would be

793
00:30:16,159 --> 00:30:19,440
incentivized to actually do the

794
00:30:17,820 --> 00:30:22,260
replication because it would make the

795
00:30:19,440 --> 00:30:25,919
papers much more valuable and people

796
00:30:22,260 --> 00:30:28,260
would would trust them and it the fact

797
00:30:25,919 --> 00:30:30,899
that these papers are in the end used

798
00:30:28,260 --> 00:30:33,240
for tenure cases for appointments people

799
00:30:30,899 --> 00:30:36,179
have a very strong incentive to make

800
00:30:33,240 --> 00:30:38,399
them as credible as as possible so I

801
00:30:36,179 --> 00:30:40,020
think it's sort of clear that we need we

802
00:30:38,399 --> 00:30:43,919
need to go there

803
00:30:40,020 --> 00:30:46,500
and yes it does you know even without

804
00:30:43,919 --> 00:30:49,200
the cost of of the replication itself

805
00:30:46,500 --> 00:30:51,720
being put on the authors making the code

806
00:30:49,200 --> 00:30:54,299
and the data available in that way puts

807
00:30:51,720 --> 00:30:57,779
a cost on the authors but again I think

808
00:30:54,299 --> 00:31:00,240
that's that's fine we've moved over the

809
00:30:57,779 --> 00:31:03,179
kind of over the longer term

810
00:31:00,240 --> 00:31:06,720
to a culture where people publish more

811
00:31:03,179 --> 00:31:09,240
and more papers uh I think improving the

812
00:31:06,720 --> 00:31:10,760
quality rather than the the quantity of

813
00:31:09,240 --> 00:31:13,500
the papers

814
00:31:10,760 --> 00:31:16,679
there would be

815
00:31:13,500 --> 00:31:19,080
be addressing that change a little bit

816
00:31:16,679 --> 00:31:21,419
and anything we can do to improve the

817
00:31:19,080 --> 00:31:24,899
quality of what we actually published I

818
00:31:21,419 --> 00:31:27,539
think I think is incredibly important

819
00:31:24,899 --> 00:31:29,520
I'm sorry the

820
00:31:27,539 --> 00:31:31,500
I feel that some of these arguments are

821
00:31:29,520 --> 00:31:34,559
a little bit the same as the arguments

822
00:31:31,500 --> 00:31:37,140
against uh pre-analysis plans where

823
00:31:34,559 --> 00:31:39,539
where people sometimes are concerned

824
00:31:37,140 --> 00:31:41,940
that this limits what they can do and it

825
00:31:39,539 --> 00:31:44,700
puts constraints on people but if you

826
00:31:41,940 --> 00:31:46,919
made it voluntary and and you saw that

827
00:31:44,700 --> 00:31:48,539
some papers did have pre-analysis plans

828
00:31:46,919 --> 00:31:52,039
and followed them they would make those

829
00:31:48,539 --> 00:31:54,419
papers more valuable so so very quickly

830
00:31:52,039 --> 00:31:57,240
experiments without pre-analysis plans

831
00:31:54,419 --> 00:31:59,960
would be viewed as a second rate and so

832
00:31:57,240 --> 00:32:02,760
it's the this is a way of of

833
00:31:59,960 --> 00:32:06,120
distinguishing high quality papers from

834
00:32:02,760 --> 00:32:09,419
from lower quality papers and I my guess

835
00:32:06,120 --> 00:32:11,220
is that that fairly that over the next

836
00:32:09,419 --> 00:32:13,740
10 years pretty much all the journals

837
00:32:11,220 --> 00:32:16,760
are going to assist on on replicating

838
00:32:13,740 --> 00:32:18,960
things one way or the other there's a no

839
00:32:16,760 --> 00:32:21,320
it's just going to send a big quality

840
00:32:18,960 --> 00:32:21,320
signal

841
00:32:23,159 --> 00:32:31,020
then well so on the issue of

842
00:32:27,600 --> 00:32:32,580
how far to take the uh code and data

843
00:32:31,020 --> 00:32:35,279
requirements because that is an issue

844
00:32:32,580 --> 00:32:37,500
because not everyone has the budget to

845
00:32:35,279 --> 00:32:38,520
higher replication teams and things like

846
00:32:37,500 --> 00:32:41,220
that

847
00:32:38,520 --> 00:32:45,059
um so what uh what we've done is I have

848
00:32:41,220 --> 00:32:47,580
a data editor who's responsible that to

849
00:32:45,059 --> 00:32:49,860
verify that all the archives submitted

850
00:32:47,580 --> 00:32:52,620
match our requirements he doesn't run

851
00:32:49,860 --> 00:32:54,480
the code but he looks at it to verify

852
00:32:52,620 --> 00:32:56,100
that one could run the code if you have

853
00:32:54,480 --> 00:32:58,620
all the things and that it's well

854
00:32:56,100 --> 00:33:01,500
documented enough so that someone could

855
00:32:58,620 --> 00:33:04,080
do it which is still a substantial

856
00:33:01,500 --> 00:33:07,020
investment of time but it's what we're

857
00:33:04,080 --> 00:33:10,320
trying to do as a interim solution

858
00:33:07,020 --> 00:33:12,240
between not requiring anything

859
00:33:10,320 --> 00:33:16,159
letting people put up whatever they feel

860
00:33:12,240 --> 00:33:16,159
like and to the full

861
00:33:16,519 --> 00:33:21,600
checking that Lars does different

862
00:33:20,039 --> 00:33:23,880
journals you know will have different

863
00:33:21,600 --> 00:33:26,940
abilities to do these sorts of things

864
00:33:23,880 --> 00:33:29,100
and hopefully over time this gets easier

865
00:33:26,940 --> 00:33:31,740
and more journals will be able to do the

866
00:33:29,100 --> 00:33:35,480
full computational checks but it is

867
00:33:31,740 --> 00:33:38,820
definitely a costly thing to do

868
00:33:35,480 --> 00:33:40,620
note that we don't actually actually run

869
00:33:38,820 --> 00:33:43,200
the code on everything because of the

870
00:33:40,620 --> 00:33:45,360
issue of secret or typically not secret

871
00:33:43,200 --> 00:33:47,519
but at least hard to access confidential

872
00:33:45,360 --> 00:33:49,620
data as well which continues to remain

873
00:33:47,519 --> 00:33:52,019
an issue in lots of economics and I

874
00:33:49,620 --> 00:33:53,580
imagine Finance as well

875
00:33:52,019 --> 00:33:56,100
so I

876
00:33:53,580 --> 00:33:58,860
um echoing Tim I like the idea of data

877
00:33:56,100 --> 00:34:02,100
editors I'm going to try to finagle a

878
00:33:58,860 --> 00:34:04,919
way to get that I don't I don't have the

879
00:34:02,100 --> 00:34:08,159
budget to do what Lars does

880
00:34:04,919 --> 00:34:12,060
but I think that having somebody besides

881
00:34:08,159 --> 00:34:14,339
me go through and make sure that the

882
00:34:12,060 --> 00:34:17,339
data and code that have been disclosed

883
00:34:14,339 --> 00:34:19,440
look like they could function is really

884
00:34:17,339 --> 00:34:20,940
important because I've run into a lot of

885
00:34:19,440 --> 00:34:23,580
data and code at the journals that

886
00:34:20,940 --> 00:34:25,619
doesn't that does not function with no

887
00:34:23,580 --> 00:34:27,720
readme files with a data set that has

888
00:34:25,619 --> 00:34:31,080
one observation I think that it's

889
00:34:27,720 --> 00:34:33,839
important to make sure if if one doesn't

890
00:34:31,080 --> 00:34:37,339
have the budget to make sure that that

891
00:34:33,839 --> 00:34:39,240
at least some basic

892
00:34:37,339 --> 00:34:40,500
requirements are met

893
00:34:39,240 --> 00:34:42,839
yeah

894
00:34:40,500 --> 00:34:44,760
and I also completely disagree with all

895
00:34:42,839 --> 00:34:47,460
the the um

896
00:34:44,760 --> 00:34:50,060
arguments against I just wanted to put

897
00:34:47,460 --> 00:34:50,060
them out there

898
00:34:51,060 --> 00:34:53,099
um

899
00:34:51,780 --> 00:34:55,080
so

900
00:34:53,099 --> 00:34:56,879
I think one of the other questions and

901
00:34:55,080 --> 00:34:58,859
this relates to a few questions we have

902
00:34:56,879 --> 00:35:04,200
in the panel to sort of uh preempt some

903
00:34:58,859 --> 00:35:06,480
of those is since we can't verify all uh

904
00:35:04,200 --> 00:35:07,859
replication compendia

905
00:35:06,480 --> 00:35:09,359
um either systematically because we

906
00:35:07,859 --> 00:35:11,460
don't have General resources or

907
00:35:09,359 --> 00:35:14,640
practically because we might not in a

908
00:35:11,460 --> 00:35:17,339
timely fashion have access to the data

909
00:35:14,640 --> 00:35:21,420
what should we do what do your journals

910
00:35:17,339 --> 00:35:23,820
intend to do when you get evidence of uh

911
00:35:21,420 --> 00:35:26,099
radical incompleteness or factual

912
00:35:23,820 --> 00:35:28,980
incompleteness or factual incorrectness

913
00:35:26,099 --> 00:35:30,540
of the analysis that was deposited so

914
00:35:28,980 --> 00:35:33,240
you mentioned you can read through and

915
00:35:30,540 --> 00:35:35,400
it seems plausible but does it actually

916
00:35:33,240 --> 00:35:36,780
yield the results in the paper you can

917
00:35:35,400 --> 00:35:38,520
only check if you actually get somebody

918
00:35:36,780 --> 00:35:39,839
to run the code so at some point in time

919
00:35:38,520 --> 00:35:41,520
somebody's going to come now that you

920
00:35:39,839 --> 00:35:43,260
will have replication packages I'm going

921
00:35:41,520 --> 00:35:45,180
to say this doesn't actually work what

922
00:35:43,260 --> 00:35:49,140
do you do then

923
00:35:45,180 --> 00:35:50,760
Tony first so that hasn't come up

924
00:35:49,140 --> 00:35:53,160
um other things have come up other

925
00:35:50,760 --> 00:35:55,680
things because I only have nine yeah

926
00:35:53,160 --> 00:35:57,839
that's right that's why I'm saying you

927
00:35:55,680 --> 00:36:01,079
can describe it as sort of in the in the

928
00:35:57,839 --> 00:36:03,660
void right as a theoretical concept

929
00:36:01,079 --> 00:36:05,940
no so this is interesting

930
00:36:03,660 --> 00:36:08,099
um I'm not I'm not sure what I would do

931
00:36:05,940 --> 00:36:10,859
and my co-editors and I have discussed

932
00:36:08,099 --> 00:36:13,079
this and so we have been

933
00:36:10,859 --> 00:36:15,359
um the jfe has dealt with other things

934
00:36:13,079 --> 00:36:16,980
like potential plagiarism cases which is

935
00:36:15,359 --> 00:36:19,079
pretty straightforward to deal with you

936
00:36:16,980 --> 00:36:20,660
do a big investigation you decide one

937
00:36:19,079 --> 00:36:23,940
way or the other

938
00:36:20,660 --> 00:36:27,000
for that I'm not sure what we would do I

939
00:36:23,940 --> 00:36:28,980
would probably try to ask the elsevier

940
00:36:27,000 --> 00:36:31,160
folks for money to hire a research

941
00:36:28,980 --> 00:36:31,160
assistant

942
00:36:31,320 --> 00:36:35,420
and then to verify whether that's

943
00:36:32,880 --> 00:36:37,320
actually right yes Yeah

944
00:36:35,420 --> 00:36:38,520
Tim

945
00:36:37,320 --> 00:36:41,460
so

946
00:36:38,520 --> 00:36:44,640
um like Tony we don't have a lot of uh

947
00:36:41,460 --> 00:36:46,380
cases yet but this is one of the things

948
00:36:44,640 --> 00:36:48,359
that bedeviled me when I was trying to

949
00:36:46,380 --> 00:36:51,060
write up the policies because as someone

950
00:36:48,359 --> 00:36:52,619
who is a mechanism designer I want to

951
00:36:51,060 --> 00:36:54,119
describe all of the things that can

952
00:36:52,619 --> 00:36:57,060
possibly happen from this when I'm

953
00:36:54,119 --> 00:37:00,359
setting up rules and I don't know

954
00:36:57,060 --> 00:37:01,980
basically it's now left as well editor's

955
00:37:00,359 --> 00:37:04,560
discretion the editor will deal with it

956
00:37:01,980 --> 00:37:07,680
in terms of what the editor will do it

957
00:37:04,560 --> 00:37:10,320
depends upon how big of a violation

958
00:37:07,680 --> 00:37:13,380
um I mean you know there's always the

959
00:37:10,320 --> 00:37:15,240
possibility of

960
00:37:13,380 --> 00:37:18,599
um of withdrawing the paper in extreme

961
00:37:15,240 --> 00:37:20,099
cases if it's just a case of well the

962
00:37:18,599 --> 00:37:22,380
one of the figures doesn't quite

963
00:37:20,099 --> 00:37:24,960
reproduce properly

964
00:37:22,380 --> 00:37:26,400
and I'll just send the authors a polite

965
00:37:24,960 --> 00:37:29,880
email saying you might want to update it

966
00:37:26,400 --> 00:37:32,400
but if it looks like they've uh really

967
00:37:29,880 --> 00:37:34,500
done something bad then you know one has

968
00:37:32,400 --> 00:37:38,119
to have the more Extreme Measures on the

969
00:37:34,500 --> 00:37:40,920
table it's just it's difficult to

970
00:37:38,119 --> 00:37:43,859
specify the exact procedures and the

971
00:37:40,920 --> 00:37:46,619
remedies to them in advance in my view

972
00:37:43,859 --> 00:37:48,780
um I I'm not ready to do that yet until

973
00:37:46,619 --> 00:37:51,440
I have some actual cases where there's

974
00:37:48,780 --> 00:37:51,440
been an issue

975
00:37:51,599 --> 00:37:59,099
we do whatever you guys thought about it

976
00:37:55,740 --> 00:38:02,960
well so we we are going to move to to

977
00:37:59,099 --> 00:38:06,240
replicating things so completely and my

978
00:38:02,960 --> 00:38:08,460
experience so far when we just had

979
00:38:06,240 --> 00:38:12,540
graduate students do some applications

980
00:38:08,460 --> 00:38:15,359
is that most of the that few of the

981
00:38:12,540 --> 00:38:18,000
papers directly replicate mostly because

982
00:38:15,359 --> 00:38:21,000
authors are are just not really very

983
00:38:18,000 --> 00:38:23,820
skilled in putting together replication

984
00:38:21,000 --> 00:38:26,820
packages they don't really try them out

985
00:38:23,820 --> 00:38:28,160
and so there's some files missing they

986
00:38:26,820 --> 00:38:31,140
don't

987
00:38:28,160 --> 00:38:34,260
things are updated in a way that the

988
00:38:31,140 --> 00:38:35,760
whole set of programs doesn't run so I'm

989
00:38:34,260 --> 00:38:36,900
actually very hopeful that in the long

990
00:38:35,760 --> 00:38:38,820
run

991
00:38:36,900 --> 00:38:41,700
we're going to get to a situation you

992
00:38:38,820 --> 00:38:45,119
know when people become more fluent in

993
00:38:41,700 --> 00:38:48,420
person control and and using get GitHub

994
00:38:45,119 --> 00:38:50,700
that we get get it a much better state

995
00:38:48,420 --> 00:38:52,440
of the world where things will replicate

996
00:38:50,700 --> 00:38:56,579
much faster

997
00:38:52,440 --> 00:38:57,960
but you know I I agree with Timmy if we

998
00:38:56,579 --> 00:39:00,740
you know currently we'll still be

999
00:38:57,960 --> 00:39:04,560
publishing papers that are not um

1000
00:39:00,740 --> 00:39:07,140
verified yet and if it turns out that

1001
00:39:04,560 --> 00:39:09,839
but we do require people to postcode and

1002
00:39:07,140 --> 00:39:12,119
if it turns out that the code doesn't

1003
00:39:09,839 --> 00:39:15,599
replicate the things that we published

1004
00:39:12,119 --> 00:39:17,160
then we could do anything from just

1005
00:39:15,599 --> 00:39:19,500
updating the code because they missed

1006
00:39:17,160 --> 00:39:23,099
some files or if the results really

1007
00:39:19,500 --> 00:39:25,380
don't uh don't don't seem if it turns

1008
00:39:23,099 --> 00:39:27,960
out it's not really possible to to get

1009
00:39:25,380 --> 00:39:30,359
to the results in any serious way then

1010
00:39:27,960 --> 00:39:33,780
then we could withdraw the paper

1011
00:39:30,359 --> 00:39:36,420
and uh that was something I was

1012
00:39:33,780 --> 00:39:38,700
concerned with at some point so now in

1013
00:39:36,420 --> 00:39:42,119
the language when you actually submit a

1014
00:39:38,700 --> 00:39:43,920
paper you make clear authors submitters

1015
00:39:42,119 --> 00:39:48,240
and acknowledge that we do have the

1016
00:39:43,920 --> 00:39:50,700
right to to withdraw papers exposed

1017
00:39:48,240 --> 00:39:54,000
if if we think they're not there's

1018
00:39:50,700 --> 00:39:56,400
mistakes in there and so that would be

1019
00:39:54,000 --> 00:39:58,740
the the ultimate sanction or we could

1020
00:39:56,400 --> 00:40:02,280
publish Corrections if we think that the

1021
00:39:58,740 --> 00:40:05,220
results more or less hold up but but

1022
00:40:02,280 --> 00:40:07,800
there are important changes when when

1023
00:40:05,220 --> 00:40:09,720
people try to to verify things but I

1024
00:40:07,800 --> 00:40:12,359
really would like to get beyond that and

1025
00:40:09,720 --> 00:40:15,780
just not publish papers until they can

1026
00:40:12,359 --> 00:40:18,660
can get the replicated

1027
00:40:15,780 --> 00:40:20,280
well to sort of throw in as sort of my

1028
00:40:18,660 --> 00:40:22,560
experience from what I you know

1029
00:40:20,280 --> 00:40:24,720
flippantly call experience from a

1030
00:40:22,560 --> 00:40:26,339
thousand papers you are going to find

1031
00:40:24,720 --> 00:40:27,720
all sorts of cases where even if you

1032
00:40:26,339 --> 00:40:29,460
have all the code in the data the

1033
00:40:27,720 --> 00:40:31,440
analysis runs for three months are you

1034
00:40:29,460 --> 00:40:33,180
willing to wait that amount of time to

1035
00:40:31,440 --> 00:40:35,460
actually run it right you're going to

1036
00:40:33,180 --> 00:40:37,740
run into lots of cases with data where

1037
00:40:35,460 --> 00:40:39,599
it might run on the synthetic data that

1038
00:40:37,740 --> 00:40:41,339
you might require as part of the process

1039
00:40:39,599 --> 00:40:43,320
but it might not actually run on the

1040
00:40:41,339 --> 00:40:45,180
confidential data because they didn't

1041
00:40:43,320 --> 00:40:47,640
rerun it because it's difficult to

1042
00:40:45,180 --> 00:40:49,079
access and things like that so we're you

1043
00:40:47,640 --> 00:40:51,180
are going to run into all of those and I

1044
00:40:49,079 --> 00:40:52,619
I can't sort of say uh we have a

1045
00:40:51,180 --> 00:40:55,260
solution because I'm not mechanism

1046
00:40:52,619 --> 00:40:57,839
designed complete by Tim's Criterion we

1047
00:40:55,260 --> 00:41:00,180
have a a policy for small changes so you

1048
00:40:57,839 --> 00:41:02,280
can revise your replication package and

1049
00:41:00,180 --> 00:41:04,260
I reach out to authors on a regular

1050
00:41:02,280 --> 00:41:07,320
basis for that we have not yet

1051
00:41:04,260 --> 00:41:08,940
encountered a hard case where we could

1052
00:41:07,320 --> 00:41:10,440
not verify and the verifiers who

1053
00:41:08,940 --> 00:41:12,300
ultimately get access to the data show

1054
00:41:10,440 --> 00:41:14,280
that everything's wrong in the paper we

1055
00:41:12,300 --> 00:41:17,660
have not yet found that but that's just

1056
00:41:14,280 --> 00:41:17,660
a matter of time for it to appear

1057
00:41:18,619 --> 00:41:24,839
yeah probably what I would like to do is

1058
00:41:21,540 --> 00:41:27,180
is have a footnote somewhere in the

1059
00:41:24,839 --> 00:41:29,640
paper or a post script where we say you

1060
00:41:27,180 --> 00:41:32,280
know this is the state of of replication

1061
00:41:29,640 --> 00:41:34,680
for this paper where either we sign off

1062
00:41:32,280 --> 00:41:36,119
saying we replicated all the numbers in

1063
00:41:34,680 --> 00:41:38,460
this paper or

1064
00:41:36,119 --> 00:41:39,900
there were you know there's some

1065
00:41:38,460 --> 00:41:42,780
difficulties

1066
00:41:39,900 --> 00:41:44,700
at a to the point that we decided to to

1067
00:41:42,780 --> 00:41:47,760
stop the process

1068
00:41:44,700 --> 00:41:50,220
because if it took three months we're

1069
00:41:47,760 --> 00:41:52,260
not going to do that or the data we're

1070
00:41:50,220 --> 00:41:53,579
confidential and so we can't reproduce

1071
00:41:52,260 --> 00:41:57,119
it

1072
00:41:53,579 --> 00:41:59,339
but here here's what what the reader

1073
00:41:57,119 --> 00:42:01,440
would like to replicate these results

1074
00:41:59,339 --> 00:42:03,900
should know and that would just be part

1075
00:42:01,440 --> 00:42:06,060
of the the record for the paper I'll

1076
00:42:03,900 --> 00:42:07,980
note that note about success or an art

1077
00:42:06,060 --> 00:42:10,260
of reproducibility as part of the

1078
00:42:07,980 --> 00:42:13,619
manuscripts and The Economic Journal so

1079
00:42:10,260 --> 00:42:17,400
um does put that into their format yes I

1080
00:42:13,619 --> 00:42:20,460
I think that that same that makes sense

1081
00:42:17,400 --> 00:42:23,339
let me uh uh turn to some of the

1082
00:42:20,460 --> 00:42:26,460
questions from the audience maybe

1083
00:42:23,339 --> 00:42:29,160
um so one question that that has come up

1084
00:42:26,460 --> 00:42:31,440
um that I think relates back to what a

1085
00:42:29,160 --> 00:42:33,000
few of you have said about this kind of

1086
00:42:31,440 --> 00:42:36,000
needs to flow into our graduate

1087
00:42:33,000 --> 00:42:37,800
education and replications are actually

1088
00:42:36,000 --> 00:42:39,300
done out there

1089
00:42:37,800 --> 00:42:40,800
um I mean that was part of our initial

1090
00:42:39,300 --> 00:42:42,240
observation as well is that there's a

1091
00:42:40,800 --> 00:42:43,980
lot of replications going on but nobody

1092
00:42:42,240 --> 00:42:45,900
knows about them

1093
00:42:43,980 --> 00:42:49,079
um so from two uh panelists Stephen

1094
00:42:45,900 --> 00:42:52,560
Eglin and Fernando asses uh the question

1095
00:42:49,079 --> 00:42:54,000
about to crowdsource reproductions in

1096
00:42:52,560 --> 00:42:56,220
some fashion there's a variety of

1097
00:42:54,000 --> 00:42:57,839
resources out there uh the replication

1098
00:42:56,220 --> 00:42:59,280
Wiki has been around for a while the

1099
00:42:57,839 --> 00:43:00,660
social science reproduction platform

1100
00:42:59,280 --> 00:43:03,060
that I've been involved with is out

1101
00:43:00,660 --> 00:43:05,339
there to record both positive and

1102
00:43:03,060 --> 00:43:07,680
negative replications

1103
00:43:05,339 --> 00:43:09,180
as a minimal Outlet there's journals

1104
00:43:07,680 --> 00:43:11,819
that will accept comments about

1105
00:43:09,180 --> 00:43:13,440
successful or unsuccessful replications

1106
00:43:11,819 --> 00:43:15,660
as well

1107
00:43:13,440 --> 00:43:17,460
um but thinking of this as a graduate

1108
00:43:15,660 --> 00:43:19,800
student activity

1109
00:43:17,460 --> 00:43:22,440
um is this something where we need to

1110
00:43:19,800 --> 00:43:24,839
move as a discipline to sort of more

1111
00:43:22,440 --> 00:43:26,280
strongly uh record all of these things

1112
00:43:24,839 --> 00:43:27,960
that are going on all over the place and

1113
00:43:26,280 --> 00:43:30,440
incorporate it as part of our graduate

1114
00:43:27,960 --> 00:43:30,440
education

1115
00:43:30,720 --> 00:43:35,780
so I will say oh well let's start with

1116
00:43:33,300 --> 00:43:35,780
her exactly

1117
00:43:35,839 --> 00:43:39,960
I was nodding

1118
00:43:38,220 --> 00:43:42,060
um

1119
00:43:39,960 --> 00:43:43,380
I think I'm the only empiricist in the

1120
00:43:42,060 --> 00:43:45,060
panel

1121
00:43:43,380 --> 00:43:47,040
um

1122
00:43:45,060 --> 00:43:48,960
so one thing that I have moved to is

1123
00:43:47,040 --> 00:43:50,760
when I co-author with graduate students

1124
00:43:48,960 --> 00:43:55,200
I've been using GitHub for about four

1125
00:43:50,760 --> 00:43:58,819
years now is I teach them about Version

1126
00:43:55,200 --> 00:44:02,400
Control and making sure that you can

1127
00:43:58,819 --> 00:44:05,099
do everything that it that somebody else

1128
00:44:02,400 --> 00:44:07,740
could do exactly what you did

1129
00:44:05,099 --> 00:44:10,859
I also have all of the stuff in my

1130
00:44:07,740 --> 00:44:12,240
papers double coded so that that to try

1131
00:44:10,859 --> 00:44:14,280
to make sure that there aren't mistakes

1132
00:44:12,240 --> 00:44:17,700
and I think that

1133
00:44:14,280 --> 00:44:20,220
um having training like that that is not

1134
00:44:17,700 --> 00:44:23,640
just from an advisor to an advisee but

1135
00:44:20,220 --> 00:44:28,099
perhaps part of the applied econometrics

1136
00:44:23,640 --> 00:44:28,099
class is a really good idea

1137
00:44:29,760 --> 00:44:33,839
um just to add to that idea

1138
00:44:32,099 --> 00:44:35,940
um we've actually done sort of peer

1139
00:44:33,839 --> 00:44:37,920
evaluation of code within classes a few

1140
00:44:35,940 --> 00:44:39,359
times as well so I really like the

1141
00:44:37,920 --> 00:44:41,220
effect of that as well as part of the

1142
00:44:39,359 --> 00:44:42,900
class and

1143
00:44:41,220 --> 00:44:44,400
um I noticed that in the chat there was

1144
00:44:42,900 --> 00:44:46,260
a comment from Maria Jones at the World

1145
00:44:44,400 --> 00:44:49,020
Bank my understanding is that they also

1146
00:44:46,260 --> 00:44:50,400
do some of this on a sporadic basis that

1147
00:44:49,020 --> 00:44:52,020
there's peer review within the

1148
00:44:50,400 --> 00:44:54,420
institution of the code so I think

1149
00:44:52,020 --> 00:44:57,660
that's an excellent idea

1150
00:44:54,420 --> 00:44:59,880
Tim so I was going to point out that

1151
00:44:57,660 --> 00:45:01,980
when we were discussing how to do the

1152
00:44:59,880 --> 00:45:04,619
reviewed economic inquiry people brought

1153
00:45:01,980 --> 00:45:08,160
up the idea of putting it out there for

1154
00:45:04,619 --> 00:45:09,720
graduate students to uh review and the

1155
00:45:08,160 --> 00:45:11,099
idea of that I think is fine it's just

1156
00:45:09,720 --> 00:45:13,260
it's difficult to do it on the

1157
00:45:11,099 --> 00:45:15,960
systematic and timely basis so to take

1158
00:45:13,260 --> 00:45:18,420
that into account in

1159
00:45:15,960 --> 00:45:20,520
um publication decisions so

1160
00:45:18,420 --> 00:45:22,859
like Guido says the better idea is to

1161
00:45:20,520 --> 00:45:26,579
have it validated prior to publication

1162
00:45:22,859 --> 00:45:27,960
well unless we want three year long lags

1163
00:45:26,579 --> 00:45:29,579
you've got to get those things done

1164
00:45:27,960 --> 00:45:31,920
quickly and

1165
00:45:29,579 --> 00:45:33,900
if every Journal is asking for graduate

1166
00:45:31,920 --> 00:45:36,119
student to replicate these things that's

1167
00:45:33,900 --> 00:45:38,579
a lot of graduate student hours and it's

1168
00:45:36,119 --> 00:45:40,980
going to be idiosyncratic and it's going

1169
00:45:38,579 --> 00:45:42,660
to be difficult to manage being able to

1170
00:45:40,980 --> 00:45:44,400
put it on occasionally then you know

1171
00:45:42,660 --> 00:45:46,020
certain papers get the stamp of approval

1172
00:45:44,400 --> 00:45:49,980
and certain papers don't buy Randomness

1173
00:45:46,020 --> 00:45:53,160
I don't think that's ideal either

1174
00:45:49,980 --> 00:45:55,500
um and so in principle the idea of what

1175
00:45:53,160 --> 00:45:59,160
the model we've gone to is putting the

1176
00:45:55,500 --> 00:46:00,960
code up there that allows any graduate

1177
00:45:59,160 --> 00:46:02,460
students or anyone else to go and

1178
00:46:00,960 --> 00:46:05,339
replicate it and then come back and tell

1179
00:46:02,460 --> 00:46:07,619
us so it's spot checking in that sense

1180
00:46:05,339 --> 00:46:09,839
and it's allowing for that but it's not

1181
00:46:07,619 --> 00:46:12,119
taking it into the publication process

1182
00:46:09,839 --> 00:46:14,760
directly quite yet

1183
00:46:12,119 --> 00:46:18,060
um but but I will definitely encourage

1184
00:46:14,760 --> 00:46:21,980
graduate students to be uh you know

1185
00:46:18,060 --> 00:46:21,980
doing these replications when they can

1186
00:46:22,619 --> 00:46:25,500
Twitter you've been working with

1187
00:46:24,119 --> 00:46:27,599
graduate students as part of this was

1188
00:46:25,500 --> 00:46:30,359
that uh some structured thing or just

1189
00:46:27,599 --> 00:46:31,319
editors calling on their uh graduate

1190
00:46:30,359 --> 00:46:33,480
students who run some of the

1191
00:46:31,319 --> 00:46:35,460
replications

1192
00:46:33,480 --> 00:46:40,760
yeah I was just working with individual

1193
00:46:35,460 --> 00:46:40,760
graduate students uh on on this uh

1194
00:46:40,980 --> 00:46:44,359
on these these issues

1195
00:46:46,319 --> 00:46:48,440
um

1196
00:46:48,480 --> 00:46:52,740
yeah can I actually also answer there

1197
00:46:50,700 --> 00:46:55,619
was one question is it about why why

1198
00:46:52,740 --> 00:46:59,940
don't the Publishers pay for this so you

1199
00:46:55,619 --> 00:47:03,660
know in the end I think that's nah

1200
00:46:59,940 --> 00:47:06,960
there's costs in running a journal and

1201
00:47:03,660 --> 00:47:09,440
those are going to go up if we provide

1202
00:47:06,960 --> 00:47:12,300
more services

1203
00:47:09,440 --> 00:47:14,520
if we replicate the papers that's that's

1204
00:47:12,300 --> 00:47:16,680
going to have to come out of somewhere

1205
00:47:14,520 --> 00:47:18,720
and so so

1206
00:47:16,680 --> 00:47:21,180
they can imagine over the years

1207
00:47:18,720 --> 00:47:23,280
economics has changed publicists uh when

1208
00:47:21,180 --> 00:47:26,880
they they thought that was appropriate

1209
00:47:23,280 --> 00:47:29,819
but it's not the case that it's this

1210
00:47:26,880 --> 00:47:32,099
could just recharge to the publicist uh

1211
00:47:29,819 --> 00:47:35,640
somewhere you know the people will

1212
00:47:32,099 --> 00:47:38,040
benefit from the journals need to uh to

1213
00:47:35,640 --> 00:47:39,839
pay for for these things and we you know

1214
00:47:38,040 --> 00:47:41,819
the econometric and we work with with

1215
00:47:39,839 --> 00:47:44,160
submission fees

1216
00:47:41,819 --> 00:47:45,300
and publication fees and you could

1217
00:47:44,160 --> 00:47:48,599
imagine

1218
00:47:45,300 --> 00:47:51,060
making getting it out of the publication

1219
00:47:48,599 --> 00:47:52,700
fees but it does have to come come out

1220
00:47:51,060 --> 00:47:55,220
of somewhere

1221
00:47:52,700 --> 00:47:59,460
whether that's you know you could also

1222
00:47:55,220 --> 00:48:02,579
redirect uh reviewing resources uh but

1223
00:47:59,460 --> 00:48:03,900
it it does have to come out of somewhere

1224
00:48:02,579 --> 00:48:05,339
or you're going to just get a lot of

1225
00:48:03,900 --> 00:48:07,760
theory papers

1226
00:48:05,339 --> 00:48:11,339
yes

1227
00:48:07,760 --> 00:48:13,440
so I so when I was thinking about this I

1228
00:48:11,339 --> 00:48:15,420
was thinking about could I do this if I

1229
00:48:13,440 --> 00:48:17,220
increase the submission fee which for

1230
00:48:15,420 --> 00:48:20,460
the jfe is already pretty high because

1231
00:48:17,220 --> 00:48:23,040
we pay our referees a lot

1232
00:48:20,460 --> 00:48:25,859
um and I would have had to um have an

1233
00:48:23,040 --> 00:48:28,740
enormously High submission fee to pay

1234
00:48:25,859 --> 00:48:31,140
for somebody like Lars and to have

1235
00:48:28,740 --> 00:48:34,440
actual replications and I decided that

1236
00:48:31,140 --> 00:48:38,420
that was not going to be the best thing

1237
00:48:34,440 --> 00:48:38,420
to do so I put that aside

1238
00:48:39,000 --> 00:48:43,859
I mean if I I want to sort of put a

1239
00:48:40,980 --> 00:48:46,920
frame on it I guess uh my My overall

1240
00:48:43,859 --> 00:48:49,079
budget is about uh let's say twice what

1241
00:48:46,920 --> 00:48:51,599
a normal editor would be paid but that's

1242
00:48:49,079 --> 00:48:53,579
for all eight journals that each have at

1243
00:48:51,599 --> 00:48:55,920
least another eight co-editors so in the

1244
00:48:53,579 --> 00:48:57,960
overall budget of a journal it isn't

1245
00:48:55,920 --> 00:49:02,040
actually that much I think but I'm

1246
00:48:57,960 --> 00:49:04,560
biased um but uh you know is that part

1247
00:49:02,040 --> 00:49:06,900
of the process of doing so uh let me

1248
00:49:04,560 --> 00:49:09,240
pick another question from the audience

1249
00:49:06,900 --> 00:49:11,099
um and I think Tim started to allude to

1250
00:49:09,240 --> 00:49:13,260
this in the sense of we shouldn't really

1251
00:49:11,099 --> 00:49:15,000
worry about it in equilibrium but have

1252
00:49:13,260 --> 00:49:16,500
replication requirements affected

1253
00:49:15,000 --> 00:49:18,300
manuscript submission numbers now I

1254
00:49:16,500 --> 00:49:21,000
think for Tony that's hard to say given

1255
00:49:18,300 --> 00:49:22,619
that how how long that is

1256
00:49:21,000 --> 00:49:26,760
um

1257
00:49:22,619 --> 00:49:28,380
uh so um I I think most of you guys have

1258
00:49:26,760 --> 00:49:30,720
only started or not yet fully

1259
00:49:28,380 --> 00:49:32,760
implemented it the only data point that

1260
00:49:30,720 --> 00:49:35,400
I have is from the early days of

1261
00:49:32,760 --> 00:49:37,140
implementing the looser uh data

1262
00:49:35,400 --> 00:49:39,420
availability policies that the AAA

1263
00:49:37,140 --> 00:49:42,420
implemented and then many other journals

1264
00:49:39,420 --> 00:49:44,940
relatively quickly picked up on there is

1265
00:49:42,420 --> 00:49:46,920
some evidence that the one Journal that

1266
00:49:44,940 --> 00:49:49,859
didn't for a while had a bit of a surge

1267
00:49:46,920 --> 00:49:51,000
of uh submissions but that petered out

1268
00:49:49,859 --> 00:49:53,099
after some time and I think the

1269
00:49:51,000 --> 00:49:53,940
reputation effect then kicked in that

1270
00:49:53,099 --> 00:49:56,040
that

1271
00:49:53,940 --> 00:49:58,740
um all of you have sort of recorded

1272
00:49:56,040 --> 00:50:00,599
before uh so I don't I don't think that

1273
00:49:58,740 --> 00:50:02,640
so well I can have a concern for you Tim

1274
00:50:00,599 --> 00:50:05,040
I can answer a little bit because we've

1275
00:50:02,640 --> 00:50:06,359
had the policy in effect since last

1276
00:50:05,040 --> 00:50:09,240
December

1277
00:50:06,359 --> 00:50:11,040
um and I don't notice any there's no

1278
00:50:09,240 --> 00:50:13,859
there hasn't been any drop off in

1279
00:50:11,040 --> 00:50:15,119
submissions this year so I mean there's

1280
00:50:13,859 --> 00:50:16,920
always the counterfactual could there

1281
00:50:15,119 --> 00:50:18,319
have been more I don't know but Arthur

1282
00:50:16,920 --> 00:50:23,180
could have been less so it's difficult

1283
00:50:18,319 --> 00:50:23,180
causality yes believe me I have enough

1284
00:50:23,780 --> 00:50:30,599
but I I don't think we've noticed any

1285
00:50:26,760 --> 00:50:31,800
Decline and we don't really get pushed

1286
00:50:30,599 --> 00:50:34,740
back from

1287
00:50:31,800 --> 00:50:36,060
um uh authors about the policy so I

1288
00:50:34,740 --> 00:50:38,339
haven't gotten any complaints about why

1289
00:50:36,060 --> 00:50:40,859
are you demanding this from me so

1290
00:50:38,339 --> 00:50:43,440
but I I think also when they submit

1291
00:50:40,859 --> 00:50:46,800
papers expect them to be fully

1292
00:50:43,440 --> 00:50:49,020
replicable I mean nobody's is is writing

1293
00:50:46,800 --> 00:50:50,640
a paper thinking that the numbers are

1294
00:50:49,020 --> 00:50:53,520
really dodgy and they don't really know

1295
00:50:50,640 --> 00:50:56,460
where they came from they just may have

1296
00:50:53,520 --> 00:50:58,140
been optimistic in in how well they

1297
00:50:56,460 --> 00:51:00,900
would remember exactly what they did and

1298
00:50:58,140 --> 00:51:02,940
how to do the replication but

1299
00:51:00,900 --> 00:51:04,680
unless people are actually submitting

1300
00:51:02,940 --> 00:51:07,800
Friday night papers which obviously is

1301
00:51:04,680 --> 00:51:11,339
incredibly rare even there I think

1302
00:51:07,800 --> 00:51:13,380
people nobody's going to object very few

1303
00:51:11,339 --> 00:51:16,319
people are actually going to debt and so

1304
00:51:13,380 --> 00:51:18,359
I also saw no evidence of that I saw

1305
00:51:16,319 --> 00:51:20,579
much more evidence when it when we

1306
00:51:18,359 --> 00:51:23,940
instituted a page limit where people did

1307
00:51:20,579 --> 00:51:26,160
push back occasionally and and and said

1308
00:51:23,940 --> 00:51:29,040
they might submit to other journals but

1309
00:51:26,160 --> 00:51:32,880
with the with this I don't see any

1310
00:51:29,040 --> 00:51:35,940
evidence that that it affects anything

1311
00:51:32,880 --> 00:51:37,619
I have no way of telling because there

1312
00:51:35,940 --> 00:51:40,200
were so many things that changed when I

1313
00:51:37,619 --> 00:51:43,280
took over that

1314
00:51:40,200 --> 00:51:43,280
it's hard to know

1315
00:51:45,420 --> 00:51:48,359
um

1316
00:51:46,740 --> 00:51:50,460
let me try and see some other questions

1317
00:51:48,359 --> 00:51:53,359
from from

1318
00:51:50,460 --> 00:51:53,359
the thing here

1319
00:51:54,059 --> 00:51:58,500
um so one thing that came up and this

1320
00:51:56,339 --> 00:52:01,200
goes again to having

1321
00:51:58,500 --> 00:52:03,119
um outside entities sort of assure this

1322
00:52:01,200 --> 00:52:05,880
I think it intersects a bit with the uh

1323
00:52:03,119 --> 00:52:08,280
again the the idea of equity and and is

1324
00:52:05,880 --> 00:52:10,319
it going to be uh Grand funded or is it

1325
00:52:08,280 --> 00:52:14,460
just the rich people but the idea of an

1326
00:52:10,319 --> 00:52:18,180
outside entity uh certifying before you

1327
00:52:14,460 --> 00:52:20,220
even submit uh to the journal that uh

1328
00:52:18,180 --> 00:52:22,079
the code is reproducible I mean this can

1329
00:52:20,220 --> 00:52:24,300
take many forms and we're actually going

1330
00:52:22,079 --> 00:52:26,040
to uh pick up on some of these later on

1331
00:52:24,300 --> 00:52:28,500
in the webinar series about a

1332
00:52:26,040 --> 00:52:30,960
reproducibility agency because God is

1333
00:52:28,500 --> 00:52:33,839
one of those out there that uh has done

1334
00:52:30,960 --> 00:52:36,960
this uh of journal or or of University

1335
00:52:33,839 --> 00:52:39,599
specific mechanisms to do this I know of

1336
00:52:36,960 --> 00:52:41,940
a few of those that are out there

1337
00:52:39,599 --> 00:52:43,680
um they all cost money to some extent

1338
00:52:41,940 --> 00:52:45,180
but would this be something that because

1339
00:52:43,680 --> 00:52:47,160
it doesn't show up on the budget of a

1340
00:52:45,180 --> 00:52:49,559
journal would be of interest to to

1341
00:52:47,160 --> 00:52:52,020
journals in general uh question to to my

1342
00:52:49,559 --> 00:52:54,059
panelists here

1343
00:52:52,020 --> 00:52:56,040
if somebody came with a reproducibility

1344
00:52:54,059 --> 00:52:58,680
certificate from a university or from a

1345
00:52:56,040 --> 00:53:01,339
recognized agency would you take it

1346
00:52:58,680 --> 00:53:01,339
yes

1347
00:53:02,040 --> 00:53:06,599
and first I mean I would still want them

1348
00:53:04,559 --> 00:53:09,480
to publish I would still want them to

1349
00:53:06,599 --> 00:53:13,020
put the um archive up there so I'm not

1350
00:53:09,480 --> 00:53:15,180
going to take it anymore but I mean

1351
00:53:13,020 --> 00:53:17,339
I would love to see it

1352
00:53:15,180 --> 00:53:21,180
um but I think it would also take a

1353
00:53:17,339 --> 00:53:23,040
while for uh those private entities to

1354
00:53:21,180 --> 00:53:27,359
um earn the respect of the publication

1355
00:53:23,040 --> 00:53:29,280
Community because if I get a

1356
00:53:27,359 --> 00:53:30,780
um a receipt from some random place I've

1357
00:53:29,280 --> 00:53:33,059
never heard of I don't know what they do

1358
00:53:30,780 --> 00:53:34,680
to check it I don't know who they are

1359
00:53:33,059 --> 00:53:37,380
um so it's going to take a while for

1360
00:53:34,680 --> 00:53:42,140
those entities to develop a trustworthy

1361
00:53:37,380 --> 00:53:45,059
reputation which could be an issue

1362
00:53:42,140 --> 00:53:46,980
the University of Michigan set up a

1363
00:53:45,059 --> 00:53:49,020
reproducibility center would you accept

1364
00:53:46,980 --> 00:53:51,780
that

1365
00:53:49,020 --> 00:53:53,700
probably right probably yeah probably

1366
00:53:51,780 --> 00:53:56,040
but we would still want to know probably

1367
00:53:53,700 --> 00:53:58,440
what they're actually doing no that's

1368
00:53:56,040 --> 00:54:00,240
right but and and I think that it would

1369
00:53:58,440 --> 00:54:02,880
all It's always important to disclose

1370
00:54:00,240 --> 00:54:06,540
the code in if possible data yeah

1371
00:54:02,880 --> 00:54:08,640
correct so lastly we're talking about at

1372
00:54:06,540 --> 00:54:11,300
the submit the initial submission of a

1373
00:54:08,640 --> 00:54:11,300
paper yes

1374
00:54:12,180 --> 00:54:16,559
and I mean let's also say that would be

1375
00:54:13,920 --> 00:54:18,480
very costly because as you submit to

1376
00:54:16,559 --> 00:54:19,920
multiple different journals assuming

1377
00:54:18,480 --> 00:54:21,260
you're not lucky enough to get accepted

1378
00:54:19,920 --> 00:54:24,540
the first time

1379
00:54:21,260 --> 00:54:26,220
that you know the the code might change

1380
00:54:24,540 --> 00:54:28,140
and you've got to do this with every

1381
00:54:26,220 --> 00:54:30,599
submission as you change that could be

1382
00:54:28,140 --> 00:54:32,339
very costly to do it at that point so I

1383
00:54:30,599 --> 00:54:34,380
don't know that I would push towards

1384
00:54:32,339 --> 00:54:37,859
that equilibrium but in some sense it

1385
00:54:34,380 --> 00:54:40,500
would be nice well if if we get to the

1386
00:54:37,859 --> 00:54:42,260
points that people write code that is

1387
00:54:40,500 --> 00:54:45,240
directly

1388
00:54:42,260 --> 00:54:47,819
reproducible then it shouldn't really be

1389
00:54:45,240 --> 00:54:50,339
very costly I don't know how these these

1390
00:54:47,819 --> 00:54:52,920
places would work but if they charge

1391
00:54:50,339 --> 00:54:55,319
sort of by the hour and I submit

1392
00:54:52,920 --> 00:54:58,140
something that is just one pushing a

1393
00:54:55,319 --> 00:55:00,300
button and it and it replicates

1394
00:54:58,140 --> 00:55:01,319
because I I just do the coding very

1395
00:55:00,300 --> 00:55:04,760
carefully

1396
00:55:01,319 --> 00:55:08,160
then that should be very very cheap

1397
00:55:04,760 --> 00:55:10,740
I mean I think just a big step is is

1398
00:55:08,160 --> 00:55:13,559
getting people to do the coding much

1399
00:55:10,740 --> 00:55:15,480
more carefully than we all were taught

1400
00:55:13,559 --> 00:55:18,000
how to do in the past

1401
00:55:15,480 --> 00:55:19,920
I'm sure that uh Christopher and young

1402
00:55:18,000 --> 00:55:21,599
who wrote an email a Twitter message

1403
00:55:19,920 --> 00:55:23,040
about this is too late for Europe is

1404
00:55:21,599 --> 00:55:24,780
sitting there with a cognac right now

1405
00:55:23,040 --> 00:55:26,940
and and uh thinking about his pricing

1406
00:55:24,780 --> 00:55:30,180
model at cascad

1407
00:55:26,940 --> 00:55:32,520
um unfortunately we're at time uh so I I

1408
00:55:30,180 --> 00:55:34,920
unfortunately to cut off our discussion

1409
00:55:32,520 --> 00:55:37,500
and the many questions that are still uh

1410
00:55:34,920 --> 00:55:39,000
in the chat that we've collected there

1411
00:55:37,500 --> 00:55:40,680
um what we'll do is we'll actually send

1412
00:55:39,000 --> 00:55:43,020
around the questions we collected from

1413
00:55:40,680 --> 00:55:44,400
the chat to all the panelists and uh we

1414
00:55:43,020 --> 00:55:45,780
can address some of those in our

1415
00:55:44,400 --> 00:55:49,440
write-ups of this

1416
00:55:45,780 --> 00:55:51,720
um as as a way to do this uh so let me

1417
00:55:49,440 --> 00:55:53,280
first uh start by thanking all my

1418
00:55:51,720 --> 00:55:55,260
panelists here for having taken the time

1419
00:55:53,280 --> 00:55:57,300
for this uh very interesting discussion

1420
00:55:55,260 --> 00:56:00,300
on the topic of journals and

1421
00:55:57,300 --> 00:56:02,940
reproducibility uh thank you Guido thank

1422
00:56:00,300 --> 00:56:05,099
you Tim uh thank you Tony

1423
00:56:02,940 --> 00:56:07,800
um for all of those who joined later

1424
00:56:05,099 --> 00:56:09,900
could make it uh there we will post this

1425
00:56:07,800 --> 00:56:12,180
recording after some light editing for

1426
00:56:09,900 --> 00:56:15,180
uh to make it like slick and clean and

1427
00:56:12,180 --> 00:56:16,680
all that afterwards and uh all the

1428
00:56:15,180 --> 00:56:18,180
panelists are invited to write up a

1429
00:56:16,680 --> 00:56:20,400
small text that we will be collecting

1430
00:56:18,180 --> 00:56:22,020
over the course of the year on the uh

1431
00:56:20,400 --> 00:56:24,180
website

1432
00:56:22,020 --> 00:56:25,680
um uh Alex if you could just post the

1433
00:56:24,180 --> 00:56:27,300
web our website into the chat for

1434
00:56:25,680 --> 00:56:29,240
everybody to to see so that they can

1435
00:56:27,300 --> 00:56:33,119
reference that

1436
00:56:29,240 --> 00:56:36,059
I wanted to then invite everybody to

1437
00:56:33,119 --> 00:56:39,180
join our next webinar on October 25th

1438
00:56:36,059 --> 00:56:41,880
same time same place on reproducibility

1439
00:56:39,180 --> 00:56:45,059
and ethics irbs and Beyond I'll be

1440
00:56:41,880 --> 00:56:46,800
discussing uh the questions of data

1441
00:56:45,059 --> 00:56:48,480
acquisition of how to incorporate

1442
00:56:46,800 --> 00:56:49,680
reproducibility right from the start

1443
00:56:48,480 --> 00:56:50,880
even when you're writing consent

1444
00:56:49,680 --> 00:56:52,140
statements and all the other

1445
00:56:50,880 --> 00:56:55,200
implications when you might not even

1446
00:56:52,140 --> 00:56:57,960
have an RB with two wonderful panelists

1447
00:56:55,200 --> 00:57:00,540
then so join us again there

1448
00:56:57,960 --> 00:57:02,339
um keep sending us questions and uh

1449
00:57:00,540 --> 00:57:05,119
thank you all

1450
00:57:02,339 --> 00:57:05,119
foreign