There appear to be minor variation between these Z values and those on your chart. My guess is that this is due to rounding? Perhaps 2 dec. places didn't cut for log data.
I don't know why the z-score values are different. I calculated the residuals myself using the parameters you gave for the Verhulst model. Maybe, the way I compute the Z-score is different.
Shiraz wrote:
To check what you mean, you refer to exponential smoothing over a local window (eg prev 10 years) and then calling a shock when smoothed data at T minus smoothed data at (T-1) is greater than some threshold?
or you could test the local validity of the fit using a statistical test. A threshold on the confidence could give you the shock position.
Shiraz wrote:
To be frank, i'd really rather remove the verhulst too. It doesn't sit right with me. But as I see it, the verhulst is the only part of the model that ensures future production will return to zero. If you remove the verhulst, you need some adjustment to keep oil production finite.
It's interesting to compare your approach with the GBM approach (see http://www.peakoil.com/fortopic9205.html). The Verhulst model seems to be equivalent to the "accessibility of imitators" function in the GBM which is deformed by the control function x(t) to model the shocks. More precisely, I you take the log of the GBM model:
Code:
log(P(t))= log(qz(t)/m(m - z(t))) + log(x(t))
the first term is equivalent to the "Log-Verhulst" you used, and the second term is the shock function. Maybe, there is a way to mixed the GBM approach and yours. _________________ ______________________________________
http://GraphOilogy.blogspot.com
I calculated the residuals myself using the parameters you gave for the Verhulst model.
Here are the unabridged versions of the parameters. I can tell you the plot of the Z values looks remarkably similar to mine.
Qinf 2000
1/k 14.53091584
n 1.836780522
T1/2 2003.046951
The other thing is that i've heard excel has a minor flaw in the calculation of Standard Deviation. This never concerned me as I'm never concerned about latter decimal places. Maybe this error is worse than I thought.
Quote:
Shiraz, you can try a local smoothing technique (like LOWESS). Even the verhulst fit became better behaved.
I wonder if you could explain a bit more. I tried the lowess model you suggest, and a bunch of others, but when I fit the verhulst, the n parameter rises without reasonable bound. Do you constrain n? What kind of verhulst fits are you getting. The best thing for me with fitting log verhulst to log data is that the early data is no longer irrelevant to the fit, and this acts as an effective constraint upon n. Otherwise, correct height at circa yr 2000 completely overwhelms good shape at circa 1900-1950.
The procedure involves:
1) smoothing the data
2) Fitting using an arbitrary precision numerical algorithm (Levenberg Marquadt, inside Mathetaitca - I do not trust Excel)
3) Using the original set of data ... and constraining the number of optimization steps I generate a family of curves and look not at the Mean curves but at the the curves in the 1% -99% (parameter estimation using the Verhulst leaves a significant bias in the final estimates
4) My results (very iteration dependent ... the estimation procedure is as you know numerically unstable for this model)
Smoothed Data (this are the means though; I would like to use a Monte Carlo approach sampling from the joined probability distribution of Qinf, τ , τ1/2, n to see if I can generate a family of possible curves; needs Mathematica programming and have no time for the next 20ds)
Iter Qinf τ τ1/2 n
18 1736.2 21.5 93.06 0.2
19 2019.7 20.3 98.6 0.718
20 2303.4 19.6 103.83 1.26
21 2870.8 18.6 114.58 2.35
Looking at the curvature measures of nonlinearity, the middle two solutions are the ones most likely to be correct. When I finish a professional examination nuisance I will have more time to try a hierarchical Model looking at both discoveries and consumption. For the Verhulst my numbers is as above though _________________ "Nuclear power has long been to the Left what embryonic-stem-cell research is to the Right--irredeemably wrong and a signifier of moral weakness."Esquire Magazine,12/05
The genetic code is commaless and so are my posts.
The other thing is that i've heard excel has a minor flaw in the calculation of Standard Deviation. This never concerned me as I'm never concerned about latter decimal places. Maybe this error is worse than I thought.
Excel sucks ... there was an article a couple of years on statistical packages and add ons. Only Mathematica survived ... even Statistica was found to have errors. Most of the errors were due to instabilities of the series approximations or asymptotic formulas used in the special functions. Mathematica somehow is immune due to the arbitrary precision arithmetic (so it can choose to increase precision when the number makes no sense) and the fact that they use the "correct" analytic continuations. E.g. they compute the P values of the Student t distribution using either a 2F1 or an incomplete Beta function based on the parameters. _________________ "Nuclear power has long been to the Left what embryonic-stem-cell research is to the Right--irredeemably wrong and a signifier of moral weakness."Esquire Magazine,12/05
The genetic code is commaless and so are my posts.
The corresponding Z-score are shown in green. I computed also a pseudo Z-score based on the log of the control function composed of three shocks proposed by Guseo et al. (see thread Peak prediction based on the Riccati equation: 2007! for more details). The parameters for the GBM are the following:
We can notice that the Z-score of the control function matches closely the shock free Z-score for the period 1952-2000. Guseo did not put any shocks previous to 1951 probably because the oil infrastructure was not mature enough. After 2004, the control model goes exponential which enables to push production. Following your approach there is maybe a way to design a probabilistic GBM based on a MC simulation of the future shocks instead of assuming a determistic model like Guseo's model. _________________ ______________________________________
http://GraphOilogy.blogspot.com
have the shocks shift the verhulst locally as well
This is kind of an interesting idea. I can see each period of continuity being a fragment of its own verhulst (or other) curve with its own trajectory.
I don't know how to make it work to predict the future.
have the shocks shift the verhulst locally as well
This is kind of an interesting idea. I can see each period of continuity being a fragment of its own verhulst (or other) curve with its own trajectory.
I don't know how to make it work to predict the future.
Did you estimate URR with each local fit, or you kept that parameter fixed? _________________ "Nuclear power has long been to the Left what embryonic-stem-cell research is to the Right--irredeemably wrong and a signifier of moral weakness."Esquire Magazine,12/05
The genetic code is commaless and so are my posts.
Did you estimate URR with each local fit, or you kept that parameter fixed?
I kept all parameters (inc. URR) for the comparison curve (the verhulst) fixed. I did not use an estimation of URR besides the implicit one derived from the comparison curve.
Quote:
I can see each period of continuity being a fragment of its own verhulst (or other) curve with its own trajectory.
Yes, this is the idea. I was thinking about using a new comparison curve based upon the same parameter for U, with constant cumulative production, and passing through the post shock data point. This should generate a unique new verhulst comparison curve for each exponential regime. My problem with this is, well, 1) it is too arbitrary. too many random actions. It could fit anything. What's that famous quote... something like, "give me 4 parameters, I'll draw you an elephant, give me five, she'll be doing somersaults" 2) way too much emphasis on the post shock data-point. Why should this data-point exert such a high degree of control over the future threshold for shock?
Alternatively, the new verhulst could be created so that the log of the new verhulst was tangential to the log straight line that is to be the baseline for the next exponential regime. I hope that's kind of clear.
Also, it may be possible, somehow, as khebab suggests, to use the GBM to be the fluid verhulst, so to speak.
With regard to the GBM...
Although it's probably not clear anymore, I actually began working on this model as a reaction to the GBM. When playing around with the GBM, I noticed that exponential shocks were not only the best fitting mathematically, but also the best fits logically as well. The effect of any shock has an exponential effect into the future, because it knocks out the base upon which new production builds. Unfortunately, however, the GBM model proceeds by biasing the underlying model according to tension between a set of exponential curves. So long as the exponential behaviour of the innovators and immitiators functions overwhelms the exponential behaviour of the shocks, everything is fine. But is practice, especially with three or more shocks, what you see is the behaviour of the shocks totally overwhelming the underlying model. I think this is a property of fitting a polynomial type curve with a tension between exponentials. You know the idea about higher degree polynomials being arbitrarily good about fitting as order increases. But of course, the higher the order, then in general, the worse the fit will be outside the domain of the test data (ie. overfitting). Well, I've never heard of an equivalent idea with exponentials, but I imagine some theorem could exist whereby any polynomial of order n can be approximated by an exponential series of n terms over a finite domain. There would also be a corollary, that outside the domain, the difference between the polynomial and the exponential series is itself some exponential function.
To try to bring this back to reality, you can see what I'm getting at in Khebabs GBM chart (above). Look at the 'control model' (purple). It is an approximation of 'Z score of the GBM without shocks' (green). Notice the fit is quite nice inside the fitting domain (ie yr < 2005). But what happens outside this domain? Well we KNOW the trend in the Z score is zero. After all, they are z scores. By comparison the control model appears to rise exponentially. In my experience, this is almost always the case with GBM models. That is also the conclusion I was getting at in the previous paragraph - that ouside the fitting domain, the tension between exponential curves would resolve to an exponential difference between the original trend and the continuation of the fitted curve. The problem with the GBM for me is that the control function always exhibits this 'aberant' behaviour outside the fitting domain.
Now it may be easier to see how the model i presented earlier was in reaction to the GBM. I was trying to capture the exponential differences between shock periods without the tension between exponentials messing up the forecast. When I plotted the log of production, it was apparent that this was going to be much easier than assumed, because the underlying behaviour was exponential anyway. (therefore I could proceed by evolving two parameters, m and b, controlling 'straight lines on the log chart')
This introduced a whole bunch of new negatives, such as the relative arbitrariness of the choice of shock timing, magnitude, and new exponential regime. Recall that I resolved this by introducing with a global verhulst for 'comparison'. It looks like a dog chasing his tail, I know. I'm not sure whether in the end, I gained more or lost more, by comparison with the GBM. If i'm to be honest with myself, I have to imagine I've lost more, because there are so many arbitrary assumptions. It's just that my baby looks beautiful to me... However, if I could just find ways to cut out some of the arbitrariness...
Quote:
Excel sucks ... there was an article a couple of years on statistical packages and add ons.
Yes, but it's so easy. I like easy. I've started to learn other packages now and then, but I get impatient and go to excel to check my latest ideas. In general, errors on the order of the 8th decimal place don't bother me, and i'll accept these happily for ease of use. The only exception perhaps is non-linear fitting. This is where I prefer to use third party software because this stuff is difficult, and I don't trust excel with it.
At the end of the day, I'm doing this oil modelling stuff for fun. I don't expect my 2nd decimal place to save the world, let alone the 8th.
Another idea: if you look at the z-score of the residuals (green curve above) we can see clearly periodic fluctuations with about a 50 years period. So why not using the periodicity transform to extrapolate future shocks (see Price Prediction Based on the Periodicity Transform for details about the periodicity transform).
Here the methodology I followed:
1- in order to have a little more data in the future, I assume that the production will keep up with the population growth in order to keep a constant number of barrels per capita of about 4.5 (see thread Production Forecast with a Population Constraint for more details).
2- I fit the GBM parameters up to 2015 using URR= 2000 Gb
3- I compute the residuals in the log domain between the Bass Model (x(t)=1) which are then normalized to obtain the z-score (green curve below)
4- I apply the periodicity transform on the above z-score
5- I extend the resulting periodicity basis up to 2200 (red dashed curve below)
6- the extended z-score is then used as the control function x(t) for the GBM model
The different functions are shown below:
The two resulting GBM functions are shown below:
Using the original Guseo's shock function based on exponential shock we obtain a peak in 2017 with an URR of 1,854 Gb (black curve). This particular shock function tends toward infinity which means infinite investment to extract further oil past 2004!
The new shock function is based on an extrapolation of the observed cycles in the z-score function and is, I think, more realistic because it reaches a maximum value in 2017 which is just above the levels reached in the 70s. It suggest investments by the industry of the same order of magnitude in order to satisfy demand up to 2015. This effort collapses in 2017 producing a production peak around 2014. Interestingly, the resulting URR is smaller and is around 1,556 Gb which is near the ASPO estimate! _________________ ______________________________________
http://GraphOilogy.blogspot.com
To try to bring this back to reality, you can see what I'm getting at in Khebabs GBM chart (above). Look at the 'control model' (purple). It is an approximation of 'Z score of the GBM without shocks' (green). Notice the fit is quite nice inside the fitting domain (ie yr < 2005). But what happens outside this domain? Well we KNOW the trend in the Z score is zero. After all, they are z scores. By comparison the control model appears to rise exponentially. In my experience, this is almost always the case with GBM models. That is also the conclusion I was getting at in the previous paragraph - that ouside the fitting domain, the tension between exponential curves would resolve to an exponential difference between the original trend and the continuation of the fitted curve. The problem with the GBM for me is that the control function always exhibits this 'aberant' behaviour outside the fitting domain.
Excellent analysis! I reached the same conclusions. I'm not confrotable with the asymptotic behavior of Guseo's shock function which has no connections with reality and can lead to aberrant fit when the URR is high. The approach based on the periodicty transform produce cycles that are similar to those observed in the past in term of periodicty and amplitudes. Another question is the link between the shock function and prices fluctuations. Up to the 80s, the fluctuations seem antagonists but since the 80s it seems to be the opposite. _________________ ______________________________________
http://GraphOilogy.blogspot.com
To add to the discussion, I added the inflation adjusted crude oil price along to the different z-score curves:
Before the 80s, price spikes are correlated with negative shocks due to disruptions in production and runaway demand. In the 80s-90s, we lived from the oil glut created from the previous initial positive shocks. Prices and investments collapsed till the end of the 90s. Now it's different, prices are going up due to excessive demand and are calling for a new wave of investment in oil extraction and exploration that will probably peak before 2020. This new injection of capital will be probably enough to delay the PO date until 2014 (best case scenario) before global production will be overwhelmed by depletion. _________________ ______________________________________
http://GraphOilogy.blogspot.com
All times are GMT - 6 Hours Goto page Previous1, 2
Page 2 of 2
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
News
Discussions
Resources
Members
Forum FAQ
Search
Usergroups
Profile
Log in to check your private messages
Log in
