Political Betting and Opinion Polling

[johntel]

A thread to discuss the usefulness (or otherwise) of Betting Markets in predicting election results.

[Mr Poppy]

IMO Betting Markets are useful for... betting - should folks want to do so. Folks use different analysis models (or just guess) to make a 'prediction' of an outcome, which can obviously change with time and 'events' (eg a big move in polling or some seat specific issue that might impact certain seats). Then if someone sees the bookies odds as very different to their prediction they can choose to bet, or not.

A 'post mortem' process can then identify why the model made some mistakes, although into GE'17 then that started to occur before the results (eg Dr.Mibbles and some pollsters picking up on the 'youth quake' in Uni seats).

The better models will have %s for likelihood to win which create 'confidence intervals' around each individual seat prediction, which then scale up to a 'range' of seats for the overall result. Within the overall result lots of 'toss-ups' will fall either side of the 'toss-up' and perhaps hide trends (eg CON failing to win seats in GE'17 that they then won in GE'19)

Not a bad idea to start an Issue Specific thread for this kind of discussion but good luck keeping it off the main thread, especially into a by-election or GE 

PS I do tend to unhide posts on Issue Specific threads that I'm interested in. With a lot less comments, less partisan stuff and less need for 'groundhog days' then Issue Specific threads are quite useful IMO.

[pjw1961]

As requested by johntel, a repost of some thoughts I posted on the main thread - the main issue under discussion was whether political betting markets essentially follow opinion polls (with a side order of 'gut feeling'), which is my contention, or whether they constitute a valid separate forecasting model in line with market theory, aka 'the wisdom of crowds', as contended by some academics.

Political Betting and Opinion Polling (1 of 3)

Thanks to johntel for the reference to the academic paper. I have got hold of the full paper and read it. I have some specific observations on the paper itself, which I will include in the second post of this sequence, but reading it also crystalised some more general points on the issue at hand, which form the subject of this first post.

The paper handily reveals the philosophical basis for the proposition that betting markets are a useful indicator of political outcomes. It makes several references to what it calls the "Hayek Hypothesis" which it defines as the idea that "markets can work efficiently even when the participants' knowledge of the environment or other participants is limited". This is of course one of the dominant ideologies of the last 50 years and the authors tend to treat it as a given fact for their purpose rather than a contested economic theory.

Wikipedia notes: "In 1945, F.A. Hayek argued in his article The Use of Knowledge in Society that markets were the most effective way of aggregating the pieces of information dispersed among individuals within a society. Given the ability to profit from private information, self-interested traders are motivated to acquire and act on their private information. In doing so, traders contribute to more and more efficient market prices. In the competitive limit, market prices reflect all available information and prices can only move in response to news."

en.wikipedia.org/wiki/Efficient-market_hypothesis

This is not the place to discuss the merits or otherwise of market ideology - the wikipedia article notes some of the objections to it raised from capitalist sources. Socialists would doubtless have other critiques. Instead I will confine myself to its applicability to political betting markets.

Some of the more extreme advocates of the efficiency of markets talk as if the market mechanism itself is enough to achieve this efficient pricing (in the context of an election, the pricing being the odds of particular outcomes occurring), but we can ignore such zealotry and agree that market prices are driven by information together with a dash of irrational human emotion. So the question for political betting - as opposed to, say, the stock market or betting on horse races - is what are the sources of information available? I can think of the following:

1. The useful
(a) Advance knowledge of an exit poll - this would be really useful for 'beating the market' but by definition will only be available late on the day of the election itself. The polling companies know the value of this data and take steps to keep it confidential.
(b) Knowledge of postal voting prior to the election date - not as good as (a) but still valuable as these are real votes. As I have noted, it is a criminal offence for anyone to release this information in advance of the polls closing on election day. I have never heard of a confirmed case of it, and I doubt it occurs very often, if at all
(c) Opinion polling data - this is likely to be the main driver of changes. For example, looking at the 2010 election the betting markets had the Conservatives as firm favorites to win an overall majority but as the polls showed 'Clegg mania' and a Lib Dem serge, a lot of betting money shifted toward the Lib Dems to do well and so changed the odds. This was, of course, just the betting market following the polling. There is a caveat to (c), that these days, as well as headline voting intention, polls contain a lot of other more granular data, especially the big MRPs which seek to stratify populations. Those with the capacity to mine this data might be able to 'beat the market' as Dr Mibbles did in 2017.
(d) Past electoral history of the electoral unit (polity, constituency, etc.) that is voting. This is freely available to everyone and doesn't change. It will be the basis for the starting odds when adapted for current polling, but won't drive any changes in the market as it doesn't change until after the election.

2. The not useful (IMO - reasons given)
(e) 'Insider' knowledge - Of course insider trading in betting markets is frowned on and often outright illegal (inside company profit information or stories from the stable lads on how well a horse has trained for example) but market theory puts quite a lot of emphasis on the idea that the market has perfect knowledge that will factor such things into the market price. It is my contention that in political betting meaningful insider knowledge does not exist. There is no one who knows for certain what voters will do in the polling booth. Anyone who has taken part in a party political campaign will know and agree that the parties do not have reliable knowledge of how many votes they are going to get; canvass returns are just not that good. Political activists when they talk among themselves as to how a campaign is going invariably talk in subjective terms about how it feels on the doorstep. Furthermore, political campaigns suffer from confirmation bias, frequently optimistically over-estimating their own position. All this is very different from knowing that company XYZ made a £5m loss last quarter, which is 'proper' insider knowledge.
(f) 'Expert' opinion - Political pundits add to the gaiety of the nation but are worthless as predictors (and generally heavily biased too). Nate Silver wrote an excellent article on 538 that showed how inaccurate they are. So the real experts in this case are professional psephologists and pollsters, but they are notoriously reluctant to make firm predictions without heavily caveating them with 'unless this changes' or 'as things stand' type comments. For experts to be driving more efficient markets they would need to be secretly advising betting syndicates or large punters. I know of no evidence, or even suggestion, that this happens.
(g) 'Common sense' - most betting is going to herd toward the 'likeliest' outcome, and usually it will be right (a self-fulling prophesy but an unprofitable one since the bookies will be offering short odds). However all that amounts to is punters knowing points (c) and (d) - freely available information.

So to sum up, the political betting market has only one reliable source of information, fleshed out by 'guessing' what is happening. It is just a shadow reflection of opinion polling, because opinion polling is the only useful information there is. My contention is that betting markets simply reflect polling and get the 'wrong result' whenever polling does, so proving themselves no more efficient than polling, and in post 3 of 3 I will endeavour through historical examples to test whether this can be substantiated.

My second post, on the academic paper, will have to wait for tomorrow, as this one has taken a long time to write.

[pjw1961]

Political betting and Opinion Polls Part 2 of 3

In this one I'm going to address the paper johntel linked to. The first thing to say that it is an academic paper and in no way am I going to claim its methodology or conclusions are wrong. I have no doubt they are accurate, within the parameters of what they are saying. But I would question what some of the results may mean.

There are three things mentioned in the paper that I wish to bring out, because I think they are relevant to the discussion we are having on these pages. In addition I want to explore what their conclusion that opinion polls "corrected forecasts exhibit little bias but an unfavourable precision relative to prediction markets" actually means.

Since it is only fair to let anyone who is interested enough to read the full paper to do so, it can be found here:
centaur.reading.ac.uk/77452/1/us2012-polls-probs-bias-precision-v2.pdf
(Btw, apologies for the question marks that appear in some of the quotes below - when I downloaded the PDF it converted all the links in the article to question marks!)

Point 1 - the study is of an American election, specifically the Obama v Romney 2012 Presidential election. This is not inherently a problem, but there is one significant difference between US and UK polling practices. UK pollsters (reputable ones at least) are members of the British Polling Council and are expected to abide by certain standards, one of which is an assumption of unbiased and politically neutral polling - www.britishpollingcouncil.org/objects-and-rules/

This is not the quite the same in the US, although many pollsters will operate in the same manner. However, there are specifically Democrat and Republican leaning pollsters in the US and the authors of the paper have been obliged to allow for this, which has implications for their view of polling. On page 8/9 they say "However, it seems unlikely for many reasons that opinion polls are truly random samples, not least because a number of pollsters may have political leanings and because it seems plausible that herding instincts and other behavioural biases mean that voting intentions may be influenced by a knowledge of the intentions of other voters."

Now we would all agree that a particular sample in one poll may not be truly random, but we have a lot of polling companies and a lot of polls and at least partial knowledge of the methods of each pollster, as well as the margin of error for each poll. Nor do we have to worry about deliberate political bias in this country. The only thing we really need to worry about is the overall accuracy of the polls, which the polling industry itself spends much time thinking about. In short I think the UK polls have some - albeit small - advantages over their US counterparts in terms of reliability.

Point 2 - the risk of manipulation (deliberate or otherwise) of betting markets.
Polls are prone to error but are highly difficult to manipulate if carried out to professional standards. This is not true of betting markets and the paper contains such an example. The paper looks at three betting markets and notes that one ended up with significantly better odds for Romney than the other two as a result of certain large bets:

"One controversy in 2012 was the allegation that the Intrade prices for Romney and Obama were manipulated in an attempt to aid the Republican candidate, Romney. The impact of such trading can be observed in Figure ??, where Intrade consistently had the lowest price of our three prediction markets for Obama throughout the second half of 2012. Thus, it might be anticipated that Intrade would perform worse than the other markets, and possibly than the polls too."
And
"The 2012 election, and to a lesser extent the 2004 and 2008 elections, also bore witness to a distinct divergence between the two most commonly known prediction markets, Intrade and
Betfair, with Republican presidential candidates tending to be priced more favourably on Intrade than on Betfair. This divergence was noted by ?, who analysed the behaviour of one particular trader who lost around $4m, apparently in manipulating the market price in favour of the Republican candidate, Mitt Romney. A clear implication of the ? paper is that there was a distinct bias on Intrade relative to Betfair (and also to the Iowa Electronic Markets); investigating the bias forms part of our description of each market"

The paper deals with this by adjusting for it, as it does with polling bias. However, in the more fundamentalist end of market philosophy such manipulation should be self correcting and not affect price and yet it clearly was not - Intrade ended up with different pricing to the other two markets, which runs counter to the notion that markets automatically revert to the 'correct' price (i.e. to the correct odds of victory and defeat in election betting).

Point 3 - The paper acknowledges, correctly, that neither betting markets nor polls are actually predictions of results, before going on the compare their merits as predictions. Betting prices are a balance between punters and bookies trying make money; polls are a snapshot of opinion at a point in time (and so subject to change). The authors concede this before stating - again correctly - that they are used to make forecasts. Of course polling companies themselves have sometimes muddied the waters by getting into the prediction business themselves - notably with MRPs.

Nevertheless I think it is a point brushed over too lightly. Most polling companies would argue that it is only their last few polls before an election that are likely to be close to the final result, and even then they are a risk from late swing as well as the usual sampling errors. Likewise the charts in the paper for betting patterns show clearly that most bets are placed very close to the election when the result is likely becoming obvious - so their accuracy is really a self-fulfilling prophesy, i.e.

"The largest single transaction was for $93,796.87, placed at 23:45:51 UTC on election day, 6th November 2012, laying Obama to win. The second largest transaction came less than an hour
earlier and similarly was to lay Obama for sixteen pounds less than the largest trade. Both of these trades were at an average price of 1.24, which implies a probability of that event occurring of about 80%. The largest transaction to back a bet came four days previously on 2nd November at 07:04:29, and was for $88,733.56, backing Obama to win"

So you could argue that both markets and polls are only truly accurate when predicting the obvious! I think fundamentally what I are arguing against is this line of thinking:
"The theory of enlightened voters might suggest that opinion polls during election campaigns will display more variance than prediction markets, because opinion polls reflect
voters who are becoming enlightened, while prediction markets tend to be self-selected samples of those who are already enlightened regarding the likely outcome of the election"

I can see no sources that would enable gamblers to be more enlightened about the result than the voters in a poll who aren't even being asked to predict the outcome but simply how they personally will vote. The source of 'enlightenment' for the punters is the polls themselves. This was the subject of my first post.

Point 4: The paper concludes by saying "In the process of converting vote shares to probability forecasts in order to allow direct comparisons to prediction markets, we correct the polls both for a range of known biases and for unknown biases, and we found that these corrected forecasts exhibit little bias but an unfavourable precision relative to prediction markets."

This sounds like an endorsement of the superiority of markets as predictors, but what do the authors mean by "precision". They define this on page 12: "Precision refers to the spread of forecasts, independent of the true value or outcome. As such, the variance of forecasts is a measure." So what they are saying is that betting markets are more consistent, producing a lower spread of 'predicted' outcomes than opinion polls do. I am happy to concede that this is true. However, the question I would ask as an (amateur) psephologist is whether this is a good thing.

We are used on this site to the fact there is variation in polls, both between companies and from poll to poll within them. There are ways to allow for this and sift from the evidence the range of possibilities and also the median likely position. On the whole this is a healthy thing. A greater danger in polling is herding, where a particular position is thought correct and companies make methodological changes to 'fit in' (a notorious example being the late change in method one company - I think it was YouGov - made in 2017 to convert their result from hung parliament to Conservative victory territory because they didn't believe their own polling). So yes, betting markets are more consistent than polls, but I don't necessarily see that as an advantage.

This will be examined in my third post when I look at whether when the polls get results 'wrong' betting markets also do so (which is my working hypothesis, subject to testing). It may be a while before this appears, as I will need to do a lot of digging on the internet for past betting odds.

[pjw1961]

Finally, probably also worth reposting this one:

One further post on betting and opinion polls. In my ongoing search for information on past betting patterns in elections, I happened across this academic paper which amused me. Market enthusiasts, confident of the predictive capability of betting, whose experiment went wrong when betting markets failed to predict constituency results at the 2010 GE. I especially love the last sentence as they try to reconcile reality and market theory:

"What are the Odds? Using Constituency-level Betting Markets to Forecast Seat Shares in the 2010 UK General Elections - Matthew Wall,Maria Laura Sudulich &Kevin Cunningham

This article investigates methodologies for translating data from constituency betting markets in each of the UK's 650 constituencies into national-level predictions of parties' seat shares for the 2010 House of Commons election. We argue that information from betting markets is highly disaggregated (offering candidate-level predictions), adjustable throughout the campaign, and free to access – meaning that such data should be a useful resource for electoral forecasters. However, we find that constituency-market gambling data from the site Betfair.com proved to be a relatively poor basis for predicting party seat shares, and we also find evidence suggesting that the data were systematically biased in several ways. Nonetheless, we argue that future research in this area should compensate for these biases to harness the potential of constituency prediction markets for electoral forecasting."

www.tandfonline.com/doi/abs/10.1080/17457289.2011.629727

[johntel]

Oct 20, 2023 16:27:28 GMT pjw1961 said:

Finally, probably also worth reposting this one:

One further post on betting and opinion polls. In my ongoing search for information on past betting patterns in elections, I happened across this academic paper which amused me. Market enthusiasts, confident of the predictive capability of betting, whose experiment went wrong when betting markets failed to predict constituency results at the 2010 GE. I especially love the last sentence as they try to reconcile reality and market theory:

"What are the Odds? Using Constituency-level Betting Markets to Forecast Seat Shares in the 2010 UK General Elections - Matthew Wall,Maria Laura Sudulich &Kevin Cunningham

This article investigates methodologies for translating data from constituency betting markets in each of the UK's 650 constituencies into national-level predictions of parties' seat shares for the 2010 House of Commons election. We argue that information from betting markets is highly disaggregated (offering candidate-level predictions), adjustable throughout the campaign, and free to access – meaning that such data should be a useful resource for electoral forecasters. However, we find that constituency-market gambling data from the site Betfair.com proved to be a relatively poor basis for predicting party seat shares, and we also find evidence suggesting that the data were systematically biased in several ways. Nonetheless, we argue that future research in this area should compensate for these biases to harness the potential of constituency prediction markets for electoral forecasting."

www.tandfonline.com/doi/abs/10.1080/17457289.2011.629727

You may mock pj, but I actually like this idea and think the authors gave up too easily . It does rely on the the individual constituency markets being strong enough to give a reasonably accurate prediction for each one, which may well not be the case. I shall revisit it come GE time. 

[johntel]

pjw1961 I've inserted some comments of my own in red into your text below....

Political Betting and Opinion Polling (1 of 3)



Thanks to johntel for the reference to the academic paper. I have got hold of the full paper and read it. I have some specific observations on the paper itself, which I will include in the second post of this sequence, but reading it also crystalised some more general points on the issue at hand, which form the subject of this first post.



The paper handily reveals the philosophical basis for the proposition that betting markets are a useful indicator of political outcomes. It makes several references to what it calls the "Hayek Hypothesis" which it defines as the idea that "markets can work efficiently even when the participants' knowledge of the environment or other participants is limited". This is of course one of the dominant ideologies of the last 50 years and the authors tend to treat it as a given fact for their purpose rather than a contested economic theory.



Wikipedia notes: "In 1945, F.A. Hayek argued in his article The Use of Knowledge in Society that markets were the most effective way of aggregating the pieces of information dispersed among individuals within a society. Given the ability to profit from private information, self-interested traders are motivated to acquire and act on their private information. In doing so, traders contribute to more and more efficient market prices. In the competitive limit, market prices reflect all available information and prices can only move in response to news."



en.wikipedia.org/wiki/Efficient-market_hypothesis



This is not the place to discuss the merits or otherwise of market ideology - the wikipedia article notes some of the objections to it raised from capitalist sources. Socialists would doubtless have other critiques. Instead I will confine myself to its applicability to political betting markets.



I’m not sure that the Hayek Hypothesis (or any other ‘market ideologies’) are relevant tbh. The efficiency of betting markets can be tested by statistical analysis of actual events, though it’s much harder with political markets because of the lack of historical data points. For sports betting markets like horse races or football matches there’s a huge amount of easily accessible historical data to work with.



Some of the more extreme advocates of the efficiency of markets talk as if the market mechanism itself is enough to achieve this efficient pricing (in the context of an election, the pricing being the odds of particular outcomes occurring), but we can ignore such zealotry and agree that market prices are driven by information together with a dash of irrational human emotion.



Agreed. And I would add that market prices may also driven by other factors outside the specific market itself. Two examples - 1) political betting on the result of an election may be used by some as an alternative financial market - e.g. for hedging if a potential incoming government is likely to do something that will cost the bettor money. 2) emotional hedging - akin to betting against one’s own football team ‘so if we lose at least I’ll get the cost of the ticket back’.





So the question for political betting - as opposed to, say, the stock market or betting on horse races - is what are the sources of information available? I can think of the following:



1. The useful

(a) Advance knowledge of an exit poll - this would be really useful for 'beating the market' but by definition will only be available late on the day of the election itself. The polling companies know the value of this data and take steps to keep it confidential.

(b) Knowledge of postal voting prior to the election date - not as good as (a) but still valuable as these are real votes. As I have noted, it is a criminal offence for anyone to release this information in advance of the polls closing on election day. I have never heard of a confirmed case of it, and I doubt it occurs very often, if at all

(c) Opinion polling data - this is likely to be the main driver of changes. For example, looking at the 2010 election the betting markets had the Conservatives as firm favorites to win an overall majority but as the polls showed 'Clegg mania' and a Lib Dem serge, a lot of betting money shifted toward the Lib Dems to do well and so changed the odds. This was, of course, just the betting market following the polling. There is a caveat to (c), that these days, as well as headline voting intention, polls contain a lot of other more granular data, especially the big MRPs which seek to stratify populations. Those with the capacity to mine this data might be able to 'beat the market' as Dr Mibbles did in 2017. Agreed. There’s a lot of polling data, being produced at different times, from different polling companies using different methodologies, different sample sizes, measuring different things, asking different questions, with different margin of error. There’s also how opinions change between the poll date and the election date; the extent to which voters will say one thing to pollsters and then vote differently; the future actions of the "undecided" voters and those who did not respond to the survey, which constituents will or will not bother to vote; and a whole host of other intangible factors.



Skilful interpretation of the opinion polls leads to identification of betting opportunities. Expert bettors assess the polls and convert them to implied probabilities, compare them with the odds on offer and bet amounts that are big enough to change the odds and make them reflect the actual probabilities more closely.



A key skill for a successful punter is to be able to convert opinion poll percentages to implied probabilities of each party winning. This is a highly skilled activity involving quite advanced maths if it’s done properly.



Dr Mibbles didn’t ‘beat the market’, he got one prediction right. It doesn’t mean anything on it’s own, he would need to do it consistently to prove that he has really got an ‘edge’.



(d) Past electoral history of the electoral unit (polity, constituency, etc.) that is voting. This is freely available to everyone and doesn't change. It will be the basis for the starting odds when adapted for current polling, but won't drive any changes in the market as it doesn't change until after the election.



2. The not useful (IMO - reasons given)

(e) 'Insider' knowledge - Of course insider trading in betting markets is frowned on and often outright illegal (inside company profit information or stories from the stable lads on how well a horse has trained for example) but market theory puts quite a lot of emphasis on the idea that the market has perfect knowledge that will factor such things into the market price. It is my contention that in political betting meaningful insider knowledge does not exist. There is no one who knows for certain what voters will do in the polling booth. Anyone who has taken part in a party political campaign will know and agree that the parties do not have reliable knowledge of how many votes they are going to get; canvass returns are just not that good. Political activists when they talk among themselves as to how a campaign is going invariably talk in subjective terms about how it feels on the doorstep. Furthermore, political campaigns suffer from confirmation bias, frequently optimistically over-estimating their own position. All this is very different from knowing that company XYZ made a £5m loss last quarter, which is 'proper' insider knowledge. I think you’re probably right, I haven’t heard of any examples of insider trading in political betting.

(f) 'Expert' opinion - Political pundits add to the gaiety of the nation but are worthless as predictors (and generally heavily biased too). Nate Silver wrote an excellent article on 538 that showed how inaccurate they are. So the real experts in this case are professional psephologists and pollsters, Agreed these groups ought to be skilled at converting poll results into probabilities, though I doubt that in practice they really are. To do this properly requires a detailed understanding of the methodology and margin of error in the poll. but they are notoriously reluctant to make firm predictions without heavily caveating them with 'unless this changes' or 'as things stand' type comments. Yes if they don’t record and publish their predictions there’s no way of verifying how accurate they are For experts to be driving more efficient markets they would need to be secretly advising betting syndicates or large punters. I know of no evidence, or even suggestion, that this happens. Me neither

(g) 'Common sense' - most betting is going to herd toward the 'likeliest' outcome, and usually it will be right (a self-fulling prophesy but an unprofitable one Not unprofitable if the bet is placed at odds which are better than the true proability since the bookies will be offering short odds). However all that amounts to is punters knowing points (c) and (d) - freely available information. As before - it's one thing to have access to the data but another to be able to interpret it correctly
(h) You’ve missed out the most important group - professional and semi-professional gamblers - the ‘clever money’. These people stake larger amounts that will significantly influence the odds. They have both a working knowledge of opinion polling and of how gambling markets work and are able to identify situations where the current odds don’t reflect the genuine probabilities. In doing this they cause the odds to converge with the true probability.



So to sum up, the political betting market has only one reliable source of information, fleshed out by 'guessing' what is happening. It is just a shadow reflection of opinion polling, because opinion polling is the only useful information there is. My contention is that betting markets simply reflect polling and get the 'wrong result' The markets don’t get ‘the wrong result’ - as shown by the fact that in the long run, on average, the odds (probabilities) are accurate. Your definition of ‘the wrong result’ seems to be the favourite not winning - whereas the markets give a probability to all the possible outcomes and expect them to occur in proportion to the probability. whenever polling does, so proving themselves no more efficient than polling, and in post 3 of 3 I will endeavour through historical examples to test whether this can be substantiated.

[Mr Poppy]

Stating the obvious but a bet on a future event is obviously taking a view on the news/events between now and the date of that event.

Hence EC's 'prediction' is really a 'nowcast'. IE if a GE was held last month (on last month's polling) then LAB 93% likely to win an OM
www.electoralcalculus.co.uk/prediction_main.html

Punters expect polls to narrow between now and next GE, so they price the likelihood of LAB OM at 72%
www.betfair.com/exchange/plus/politics/market/1.167249195

As time passes then you usually see 'convergence' as the likelihood of stuff that might narrow LAB's lead decreases over time. Hence whilst Betfair unfortunately don't use the X-axis as a true date line then LAB's likelihood to win OM has moved from 50-60% to >70% this year despite very little movement in polling.

If polling doesn't change between now and GE'24 then betting markets will continue to converge with 'nowcast' predictions (ie betting markets will move to 90%+)
Alternatively then if LAB's lead starts to narrow in the next 12mths then EC (et al) nowcast will start to reduce the % likelihood of LAB winning an OM (ie EC's model will move to 72%ish)

[pjw1961]

Oct 20, 2023 21:38:01 GMT johntel said:

pjw1961 I've inserted some comments of my own in red into your text below....

This is not the place to discuss the merits or otherwise of market ideology - the wikipedia article notes some of the objections to it raised from capitalist sources. Socialists would doubtless have other critiques. Instead I will confine myself to its applicability to political betting markets.

I’m not sure that the Hayek Hypothesis (or any other ‘market ideologies’) are relevant tbh. The efficiency of betting markets can be tested by statistical analysis of actual events, though it’s much harder with political markets because of the lack of historical data points. For sports betting markets like horse races or football matches there’s a huge amount of easily accessible historical data to work with.
The reason for raising the Hayek Hypothesis was that it is the specific underpinning of the academic paper, referenced more than once. Note that I am not claiming that no markets work, rather that the political betting market lacks information other than polling, so in essence it is following polling and will be as accurate or inaccurate as the polling. 

Some of the more extreme advocates of the efficiency of markets talk as if the market mechanism itself is enough to achieve this efficient pricing (in the context of an election, the pricing being the odds of particular outcomes occurring), but we can ignore such zealotry and agree that market prices are driven by information together with a dash of irrational human emotion.

Agreed. And I would add that market prices may also driven by other factors outside the specific market itself. Two examples - 1) political betting on the result of an election may be used by some as an alternative financial market - e.g. for hedging if a potential incoming government is likely to do something that will cost the bettor money. 2) emotional hedging - akin to betting against one’s own football team ‘so if we lose at least I’ll get the cost of the ticket back’.
True, but that is where I get twitchy about seeing betting odds as of the same or greater predictive value as opinion polling. With polling I can understand the underlying logic and method, the stated margin of error and factor in the past experience of polls proving inaccurate. With the 'wisdom of crowds' argument, i.e. that the sum total of all these (often irrational) bets produces a more accurate answer, that feels like believing in a sort of magic - especially with regard to political betting, where I'm pretty convinced (but need to produce evidence) that the general expected outcome is proved inaccurate as often as polling is. 

So the question for political betting - as opposed to, say, the stock market or betting on horse races - is what are the sources of information available? I can think of the following:

(c) Opinion polling data - this is likely to be the main driver of changes. For example, looking at the 2010 election the betting markets had the Conservatives as firm favorites to win an overall majority but as the polls showed 'Clegg mania' and a Lib Dem serge, a lot of betting money shifted toward the Lib Dems to do well and so changed the odds. This was, of course, just the betting market following the polling. There is a caveat to (c), that these days, as well as headline voting intention, polls contain a lot of other more granular data, especially the big MRPs which seek to stratify populations. Those with the capacity to mine this data might be able to 'beat the market' as Dr Mibbles did in 2017. Agreed. There’s a lot of polling data, being produced at different times, from different polling companies using different methodologies, different sample sizes, measuring different things, asking different questions, with different margin of error. There’s also how opinions change between the poll date and the election date; the extent to which voters will say one thing to pollsters and then vote differently; the future actions of the "undecided" voters and those who did not respond to the survey, which constituents will or will not bother to vote; and a whole host of other intangible factors.

Skilful interpretation of the opinion polls leads to identification of betting opportunities. Expert bettors assess the polls and convert them to implied probabilities, compare them with the odds on offer and bet amounts that are big enough to change the odds and make them reflect the actual probabilities more closely.

A key skill for a successful punter is to be able to convert opinion poll percentages to implied probabilities of each party winning. This is a highly skilled activity involving quite advanced maths if it’s done properly.
I agree all of that - but note it is all ultimately based on polling

Dr Mibbles didn’t ‘beat the market’, he got one prediction right. It doesn’t mean anything on it’s own, he would need to do it consistently to prove that he has really got an ‘edge’.
Actually, that's not right. Dr Mibbles forecast a number of specific constituencies against the prevailing wisdom in 2017, Canterbury was just the most discussed. Rumour had it that he won quite a bit of money in 2017 betting on a number of constituency results where the bookies were offering odds based on past electoral history and not seeing the implications of demographic voting patterns. So he did have an edge, but it was essentially the same one the big MRP had over traditional polling in 2017, when it forecast a hung parliament. MRPs performed much less impressively in 2019 and so, it is said, did Dr Mibbles, so it was something of a one-off. 


2. The not useful (IMO - reasons given)

(e) 'Insider' knowledge - Of course insider trading in betting markets is frowned on and often outright illegal (inside company profit information or stories from the stable lads on how well a horse has trained for example) but market theory puts quite a lot of emphasis on the idea that the market has perfect knowledge that will factor such things into the market price. It is my contention that in political betting meaningful insider knowledge does not exist. There is no one who knows for certain what voters will do in the polling booth. Anyone who has taken part in a party political campaign will know and agree that the parties do not have reliable knowledge of how many votes they are going to get; canvass returns are just not that good. Political activists when they talk among themselves as to how a campaign is going invariably talk in subjective terms about how it feels on the doorstep. Furthermore, political campaigns suffer from confirmation bias, frequently optimistically over-estimating their own position. All this is very different from knowing that company XYZ made a £5m loss last quarter, which is 'proper' insider knowledge. I think you’re probably right, I haven’t heard of any examples of insider trading in political betting.
As I say, I don't so how it can exist. The waffling of political campaigns really shouldn''t be taken seriously - they don't know anything for certain. 

(f) 'Expert' opinion - Political pundits add to the gaiety of the nation but are worthless as predictors (and generally heavily biased too). Nate Silver wrote an excellent article on 538 that showed how inaccurate they are. So the real experts in this case are professional psephologists and pollsters, Agreed these groups ought to be skilled at converting poll results into probabilities, though I doubt that in practice they really are. To do this properly requires a detailed understanding of the methodology and margin of error in the poll. but they are notoriously reluctant to make firm predictions without heavily caveating them with 'unless this changes' or 'as things stand' type comments. Yes if they don’t record and publish their predictions there’s no way of verifying how accurate they are For experts to be driving more efficient markets they would need to be secretly advising betting syndicates or large punters. I know of no evidence, or even suggestion, that this happens. Me neither

(g) 'Common sense' - most betting is going to herd toward the 'likeliest' outcome, and usually it will be right (a self-fulling prophesy but an unprofitable one Not unprofitable if the bet is placed at odds which are better than the true proability Ok - unprofitable for most, otherwise bookies would not make the money they do! since the bookies will be offering short odds). However all that amounts to is punters knowing points (c) and (d) - freely available information. As before - it's one thing to have access to the data but another to be able to interpret it correctly

(h) You’ve missed out the most important group - professional and semi-professional gamblers - the ‘clever money’. These people stake larger amounts that will significantly influence the odds. They have both a working knowledge of opinion polling and of how gambling markets work and are able to identify situations where the current odds don’t reflect the genuine probabilities. In doing this they cause the odds to converge with the true probability.
There are vast number of professional gamblers who play the financial markets (although they give themselves grander titles) but are there that many in something as niche as political betting? Anyway, since they are basing their understanding of the position on polling, it doesn't really invalidate my argument. Their knowledge is only as good as the polls. 

So to sum up, the political betting market has only one reliable source of information, fleshed out by 'guessing' what is happening. It is just a shadow reflection of opinion polling, because opinion polling is the only useful information there is. My contention is that betting markets simply reflect polling and get the 'wrong result' The markets don’t get ‘the wrong result’ - as shown by the fact that in the long run, on average, the odds (probabilities) are accurate. Your definition of ‘the wrong result’ seems to be the favourite not winning - whereas the markets give a probability to all the possible outcomes and expect them to occur in proportion to the probability. whenever polling does, so proving themselves no more efficient than polling, 
You are correct about that - the shortest odds for a particular election are an statement of believed probability, with other outcomes certainly possible but just less likely. If odds were treated that way rather than as predictions I would have a lot less issue with this. However, the argument in the paper was about the predictive capability of betting and the 'prediction' will reflect the shortest odd. We had an example on our very own website in the final few days of the Mid Beds by-election of people buying into this narrative. The shortest odds switched from being a Labour win to a Conservative win (and this in the absence of new polls, so it was just guesswork by punters). This was treated by a surprising number of people as a forecast of Conservative victory (I believe they were 'evens' by the end) and there was much depression from a few posters and hostile posts about the Lib Dems costing Labour victory. It had no effect on my morale at all, any more that the various speculations supposedly issuing from the campaigns, which I also discount, and Labour duly won reasonably comfortably.

Hi Johntel. I've answered your responses to me in blue.