Is solar generation increasing? (No) The perils of cumulative totals

This graph was tweeted by Greenpeace

Which immediately brought me back to a recent Stats Chat post on cumulative totals. Cumulative totals tend to go up. And on the face of it, it doesn’t really look like growth has been accelerating recently. Growing yes, but not “skyrocketing”.

I followed up on the post, and found the original data here. Turns out it’s quite a bad graph. It’s not installed solar capacity (as the label says), but rather solar generation. Though that at least explains why the value dropped in 2012 – it’s possible that generation could decrease (less sun?), but it would seem strange that lots of people would uninstall solar panels. Worse, it turns out that they are only using real data until 2013, and the 2014 and 2015 numbers are projected.

So here is a more accurate version of the graph (dotted are estimated figures)


But if you were to draw a line through 2009-17, it doesn’t really look like exponential “skyrocketing” growth, only linear straight-line growth.

And if you plot the annual increase, not such a flash picture emerges.


Solar generation is definitely increasing (a good thing), but not skyrocketing.

Posted in Uncategorized | Leave a comment

No to faster motorway speeds.

Driving fast is fun. I’ve enjoyed Italian and French motorways at 130km/h, and the Germany Autobahn at the maximum speed attainable by the small Nissan I was driving (~190km/h).

But is it a good idea? For new motorways in New Zealand?

Assuming a car was able to travel at *exactly* the speed limit, the time savings for travelling 10km would be 33 seconds off a 6 minute trip. That doesn’t seem entirely bad. But in some sort of real world analysis (not formally published) a change of limit of 10mph led to an actual increase of only 3-4mph (author information here), so I think it is safe to assume the real world time savings might be less.

Assuming incrementally New Zealand continued to increase the quality of roads, an increased speed limit would save you 2 minutes 21 seconds from the start of the Northern Gateway to the Auckland central motorway junction (CMJ, 43km), assuming, haha, no congestion. Or all the way through to Pokeno (95km) 5min 11 sec. Or if Auckland CMJ to Hamilton was one long stretch of 110km/h. Savings 6 minutes 26 seconds. In. No. Traffic.

If that isn’t bad enough, most negative events associated with driving increase exponentially rather than linearly with increasing speed. So we have a nominal increase of 10% in the speed limit, but potentially less than 5% actual speed increase, but with more than 10% increase in fuel consumption, and a higher than 10% likely increase in crashes and fatalities.

Oh, and having vehicles travelling faster actually means less cars able to fit on the road, so more congestion.

In international contexts, truck drivers are not interested in travelling faster, realising that the miniscule time savings are more than outweighed by the increased fuel consumption.

Posted in Uncategorized | Leave a comment

More rambling on polls & bias (UPDATED)

I should clarify that my post yesterday was not intended to produce another poll of polls, but to explore differences between polling companies. As one commenter suggested, making a line more sensitive would probably be more interesting/useful for a poll of polls.

And as I eventually realised, with Roy Morgan contributing over half the poll results, it does weigh heavily on the poll of polls. With those two thoughts in mind, I’ve removed Roy Morgan from this chart, and also added a Loess (span = 0.5), which was the sensitivity used in the Wikipedia charts for the 2011 election.

As you can see, the more sensitive line does certainly vary more, and certainly counters Stuff’s headline yesterday suggesting that National were bouncing back. It’s also really obvious here just how high the Fairfax Ipsos (pink dots) is compared to most of the rest of the polls at the moment. Interestingly, at this point in the last election, it was the Roy Morgan poll that seemed to be tracking much higher for National.

This is perhaps the attraction of producing some sort of poll of polls, but trying to correct for the polling frequency, so that the average isn’t over influenced by polling frequency makes sense. I didn’t mention David Farrar’s rolling average of polls yesterday (probably because it isn’t presented as a graph), but this close to an election, he only uses the last poll, so those figures would not be influenced by that. Some discussion yesterday on this, and the suggestion that Roy Morgan ought to perhaps be overweight, as it does add more information, because it is more regular. However, perhaps an alternative would be average each polls trend. Pictured below are loess(0.5) by polling company. Interestingly, they don’t as a rule particularly follow each other, though (again in contrast to the Fairfax headline yesterday), do suggest that National’s support is softening.

Finally, back to bias, Russel Brown, on twitter, asked about how my figures would look for the smaller parties.

Fairfax have been low on NZ First (corresponding to their being high on National?). Roy Morgan again driving the trend, and with the highest estimates for NZ First.

3 News Reid Research have the highest estimates for the Conservatives overall. This is one where Roy Morgan don’t dominate, and they don’t seem to have detected the uptick in support that all the other polls have seen recently.

Not enough data points really

Definitely trending down.

Not much to say here, other than that this make’s Roy Morgan’s policy to round to 0.5 percent (and 1 percent for larger parties) really obvious, relative to the other pollsters.

So final *bias* summary

  1. Fairfax are high on National, seemingly at the expense of NZ First
  2. Roy Morgan are higher on Labour and the Greens, seemingly at the expense of National.

UPDATE: The point I meant to make this morning (but forgot)

It seems reasonable to me that there will be systematic quirks in each firms polling methodology (handily outlined here). Combined with changes over time in the population, and perhaps polling companies changing their methods (eg to try to increas response rates), there ought to be some variance attributable to the polling company. Trying to work out whose is “right” is a fool’s errand, but attempting to account and model for it seems like a good plan.


Posted in Uncategorized | Leave a comment

Thoughts on poll bias

Stuff’s headline this morning National Back on Track, got me thinking a bit more about poll bias. Partly because the Fairfax Ipsos poll seems to have much higher numbers for National, and also because of the nice work presented by Danyl with his “bias corrected” poll, Gavin White of UMR’s analysis, along with two other poll aggregators, that of a Wikipedia Editor, and also Rob Salmond at Polity.

This stuff is a little oblique to that. I’ve used the Wikipedia data scraping strategy, and then used ggplot2 in R to produce these (relatively untidy graphs). The black line represents a default loess smoother, and as well as plotting a line for each company, I’ve also plotted a default loess for each company. For National, it’s clear that (especially in recent times) Fairfax Ipsos is particularly bullish on National. Also, just note for later that the black overall fit tends to quite heavily mimic the dirty yellow line for Roy Morgan.

For Labour, what is most remarkable is how consistent (and negative) the trends are for all the lines. Note Fairfax Ipsos sticking out again recently.

The smallest party I’m going to plot is for the greens. Again note that the dirty yellow line is the same shape as the black line.

So why does the Roy Morgan profile seem to share the shape with the same overall trend? Something I’ve known, but hadn’t really considered as an influence before, is that Roy Morgan is by far the most regular and frequent poll. This means unless that is weighted out, it will always dominate the shape of the trend. And as far as I’m aware, none of the poll averaging strategies do that.

In conclusion

  1. Fairfax is a bit of an outlier (favouring National over Labour).
  2. Because Roy Morgan poll the most frequently, most attempts to aggregate over different polls will be overweight with Roy Morgan.


And for any geeks, here is my ggplot2 code

ggplot(surveys, aes(Date, National)) +
     geom_point(aes(shape = Company, colour = Company)) +
     geom_line(aes(colour = Company)) +
     geom_smooth(aes(colour = Company), method = “loess”,
     se = FALSE, lwd = 1.5) +
     geom_smooth(method = “loess”, colour = “black”, lwd = 1.5) +
    theme_grey(18, “Gill Sans MT”) +
    theme(legend.position = “bottom”, legend.text = element_text(size =10))

Posted in Uncategorized | 5 Comments

Drain Water Heat Recovery – We’ve ordered one!

Three years ago, I wrote about my discovery of drain water heat recovery (also known as grey water heat recovery). Simply put, they appear to offer the same energy savings as a solar hot water system, but for a tiny fraction of the price. I read up on it quite a bit. They are well supported overseas, and there are plenty on the market internationally. So when the opportunity arose to purchase one, I looked at the two locally available examples, and picked the one that was able to be installed vertically, the EnergyDrain. Being locally made, and cheaper also helped in the decision.

Figure 1: Figure EnergyDrain

Despite some semi-effortful attempts, in my background reading, I hadn’t come across any criticism. So I went ahead and ordered one. However, semi-fatefully, a week or so after placing the order, I mentioned this systems in a forum discussing solar water heating, and David Haywood (wearing his engineering hat) said:

I’ve done heaps of modelling on these systems and they are good in theory. The problem is the HX cost and the cleaning. Most systems use some sort of horrible draino-type stuff every few weeks.

And went on to be explicitly critical of the horizontally installed ones. The more expensive GFX that I ruled out, seems much less prone to fouling, but that is due to its vertical installation, which isn’t possible for our house.

The main problem is that the building up of scum inside the heat exchanger will reduce its efficiency; though I’m not sure if it will ever meant that efficiency reduces to zero. Anyway, we’ve bought one, it’s arriving any day now, so I think it’s worth quantifying whether we will see much in the way of savings, and whether scum build-up is an issue.

Fortuitously, once the system has been installed, we will switch our hot water cylinder onto night rate, which is separately metered, so our energy consumption for hot water will be easily measured.

In order to accurately estimate the savings, I plan to have the plumber install a bypass loop round the heat exchanger, as well as a Y-regular joint, with an inspection opening, as illustrated below

To answer the first question, what are the real-world savings, if any, I’ll use an ABBA design, as illustrated below. I’ll route the cold water through the bypass for a week, then two consecutive weeks with the heat exchanger (HXC) in operation, followed by a week with the bypass back on. The advantage of the ABBA design is that if there are any other variables changing over time, affecting our hot water energy use (eg, changes in the weather, the system getting dirtier), this should cancel it out. Efficiency can then be estimated as the ratio of energy decrease during B divided by energy used during A (here (15-10)/15 = 33%).

Then, to assess the influence of dirtiness, I can plot energy use over time, and then occasionally clean it out. My faked data below would suggest that most energy savings is lost after 90 days (dotted lines indicate cleaning, and here the efficiency declines, so kW used increases), so would suggest cleaning rather more frequently. Obviously, I secretly hope for data that does not look like this, as it would imply that I should clean it every month or two.

Obviously, I’d be much happier if it looked something like this, with the efficiency declining a bit, then plateauing.

Welcome any feedback on my design, and otherwise, I guess there will be an update in a month or two.

Posted in Uncategorized | 1 Comment

Roy Morgan predicting National landslide?

UPDATE: This post was written early yesterday (Monday), but I chickened out of posting it, because I thought it was too much of a ballsy call. However, Roy Morgan are themselves making the call, so I think it is worthy of discussion. In particular, because this is not a recent change, but part of a consistent pattern for them.

The Herald Digipoll, ONE News Colmar Brunton, Fairfax-Research International, and 3 News Reid Research are all showing support for National dropping, but Roy Morgan has National support steady, and comfortably above 50%.

Polls are invariably reported with their Margin of Error, an estimate of the precision for a party with exactly 50% support. The Margin of Error is not very useful for comparing whether there is a difference between two parties, and it certainly is no use at all for considering change over time. And change over time ought to be what we are interested in. Is support for Party X climbing or dropping?

One crude way of doing this is to add a line fitted through the datapoints, but one of the hidden aspects of variation, as I tried to illustrate on Friday is that different polls will use slightly different methods, which may mean that that poll produces consistently different results in a certain direction. As I noted then, Roy Morgan generally shows lower support for National. However, over the last month, the other polling companies have all shown a clear drop, while Roy Morgan have had National’s support climbing.

Roy Morgan sample over a week to a fortnight, which could explain them being slower to catch on to this most recent trend. Alternatively, it may be that some particular element of their survey method is producing a consistently different result for them. This is brave: if they are proved right on election night, it could be that their method is superior. If they are wrong, then they may want to reconsider how they are polling. Irrespective of the outcome, however, it brings me back to a recent summary of Daniel Kahneman’s work looking at the success of fund managers: whoever is most successful in a given year (or election?) may just be due to chance.

UPDATE 2: The downward trend exists for the other 3 polls, but because Fairfax Research International and Herald Digipoll have relatively few data points, it is hard to display in a tidy fashion. Also, Roy Morgan’s trend does not diverge in the same way for Labour and the Greens, only National.

The data is derived from wikipedia, and fit with a modified version of the graph there, using a LOESS fit, with a span value of 0.3.

Posted in Uncategorized | 1 Comment

Reading the Political Tea Leaves

Despite mentioning tea in the title, this post is about opinion polls, not the ACT of drinking tea.

There are a number of sites that maintain graphs of New Zealand political polls, notably Rob Salmond at Pundit and, curiously, Wikipedia. Wikipedia post their data in tabular form, and provide the code for R underlying the creation of their graphs, allowing anyone with certain degree of geek cred to have a hack (ie, me). Yesterday, someone on wikipedia requested an updated version focussing only on more recent polls, which I had a go at, including making the fit line a bit more sensitive to change.

Changing the sensitivity of the fit line, makes it seem like there more movement than there has been in a while, and this graph was subsequently featured on the DimPost and then the Listener.

The next question was about whether the different polling parties differentially contribute to such a rolling poll. Their sample sizes are pretty uniform, at around 800. Roy Morgan contribute over half (68 of 121) with their poll regularly conducted over several days. 3 News Reid Research have fewer polls (16) but all conducted on a single day.

But how do the different parties fare in the polls (in order of polling).

Firstly, you can really get a sense from this graphic how much more regular Roy Morgan are, and their estimates are pretty consistently low, relative to the other pollsters. 3 News Reid Research is fairly consistently high. However, in the latest few polls, they have had some lower numbers for National.

3 News Reid Research is consistently lower on Labour, which combined with the above, suggests they favour National relative to the other pollsters. The Herald Digipoll has higher values for Labour most of the time.

The Herald Digipoll and ONE News Colmar Brunton are consistently lower for the Greens than other pollsters. However, in the most recnet polls, Herald Digipoll and ONE News Colmar Brunton both show much higher numbers than usual for the Greens.


  • There is definitely variation attributable to the pollsters.
  • It does seem like there is some change, at least with the polls. Whether this translates to anything meaningful for next Saturday, who knows. I plan to add ACT and NZ First later for completeness.
Posted in Uncategorized | Leave a comment