Web Scraping in R
Posted in Web Scrapping
Comments Off on Web Scraping in R
Continuous Futures Prices in R
Introduction
Crude oil prices by delivery period define the term structure of the market. The term structure changes shape over time given shifts in price level and slope. Term structure behavior becomes clear by combining discrete futures contracts with similar maturities into a continuous time series. R code is supplied to create continuous prices by delivery period. The purpose is to show term structure behavior and to derive risk and profitability measures for oil production, marketing and trading strategies. The resulting data is tidy, well suited for model training and out-of-sample testing.
Posted in Data, R Programming
Comments Off on Continuous Futures Prices in R
Term Structure of Crude Oil 2018
An animation showing the term structure of NYMEX crude oil. For source code, go here.
Posted in Animation, Data, Economics, R Graphics
Comments Off on Term Structure of Crude Oil 2018
Plotting Forecast Data Objects Using ggplot
Robert Hyndman is the author of the forecast package in R. I’ve been using the package for long-term time series forecasts. The package comes with some built in methods for plotting forecast data objects in R that Ive wanted to customize for improved clarity and presentation. The following article achieves that goal and shares two scripts for plotting forecast data objects using ggplot.
Posted in Data Science, ggplot2, Modeling, R Programming
Comments Off on Plotting Forecast Data Objects Using ggplot
Geospatial Data and Mapping in R
Slides presented at a recent meeting of Doha R users.
Posted in Website
Comments Off on Geospatial Data and Mapping in R
From Least Squares to k-Nearest Neighbor (kNN)
The linear model is one of the most widely used data science tools and one of the most important. In contrast, there is another basic tool: the 
nearest neighbor method (NN). Prediction and classification are two uses for these models. In practice, classification results (ie. feature classes) are used by machines in many ways: to recognize faces in a crowd, to “read” road signs by distinguishing one letter from another and to set voter registration districts by separating population groups. This article applies and compares linear and non-linear classification methods
Posted in Data Science, Modeling, R Programming, Website
Comments Off on From Least Squares to k-Nearest Neighbor (kNN)
Predicting Technology Progress and Solar Growth
Technology progress is a key to solar growth and pricing. By extension, the ability to model technology progress is essential to understanding future energy supply and demand.
Solar innovation is widespread. Examples include solar cell efficiency, module manufacturing, and learning innovations with solar system installation and operation. Solar pricing and growth are also supported by innovations in enabling technology, such as battery storage, smart grids and electric vehicles.
R Functions for Best Subset Regression
Best subset regression is an technique for model building and variable selection. The method looks at all combinations of independent predictor variables for use in a multiple regression model. Model developers and analysts will often struggle with variable selection, especially when the number of predictors is high. Ideally, each set of predictors is run and the best set is selected using a criteria for model performance. The following article provides custom functions for best subset selection that are fast and easy to use.
Posted in Data Science, Faster R, Modeling
Comments Off on R Functions for Best Subset Regression
Extract Data Tables from PDF Files in R
A new method to extract data tables from PDF files is introduced. The solution combines the R programming language with the open-source Java program Tabula. The result is a convenient method that transforms documents into databases.
Benefits
The ability to train a machine to extract data tables from PDF files has several benefits:
Posted in Data, Misc Tricks, R Data Import, R Programming
Comments Off on Extract Data Tables from PDF Files in R
SpatialPoints in R: Large Data Case Study
A common task in spatial data analysis is extracting SpatialPoints inside a set of polygons or buffer zones. Analysts can use standard GIS or map tools to extract a set of points within an area of interest using manual “point-and-click” routines. This method is easy, but will probably prove impractical, especially in cases involving big data. The alternative is to train a machine to automatically extract the points in a polygon or buffer zone. This post achieves that task and presents a case-study with R code.
Posted in Data, GDAL, Spatial Analysis
Comments Off on SpatialPoints in R: Large Data Case Study
Popularity of R Programming Language
The popularity of R is rapidly increasing and is well on its way to being a top 10 programming language. The TIOBE index is a standard indicator of the popularity of all programming languages. The TIOBE index confirms that a subset of languages – those for computational statistics and data analysis – are gaining increased attention. The clear winner of the pack is the open source programming language R.
Posted in Data Science, R Programming
Comments Off on Popularity of R Programming Language
Cost Breakdown Structures for Solar PV Projects
Modules, inverters and balance of system costs define the total installed cost of a solar PV system.
The three cost components are very simple in nature. In practice, total cost is defined using a detailed cost breakdown structure. The structure must also be applied consistently across projects and over time. The result can be improved cost modeling and management.
Posted in Economics, Engineering
Comments Off on Cost Breakdown Structures for Solar PV Projects
Glare from Solar PV Modules
A common question concerning the safety of photovoltaic (PV) power systems is the impact of reflected sunlight. PV modules have the potential to impact neighboring structures or activities, notably aviation. It is important to know where the reflected light will go and what the intensity of the light will be at any point in time.
Posted in Engineering, Projects
Comments Off on Glare from Solar PV Modules
Aerosol Animation
Aerosol Optical Depth (AOD) defines the degree to which aerosols prevent the transmission of sunlight by absorption or scattering. AOD is measured using an integrated extinction coefficient over a vertical column of air. The extinction coefficient can be used to analyze solar extinction and the performance of solar power systems as a function of location and time.
Posted in Animation, Data, Modeling, R Data Import, R Graphics, Spatial Analysis
Comments Off on Aerosol Animation
Crop Raster Images in R
The maptools package has a pruneMap() function t0 crop map objects in R. In practice, the function extracts data from SpatialPolygon or SpatialLine objects given a boundary box or specific area of interest. Unfortunately, there is no equivalent function for high resolution, large data, raster images, which are common in many Earth Science applications. The following post defines a custom function to crop raster images in R and to extract data from SpatialGridDataFrames. The function is tested using a raster image from the Shuttle Radar Topography Mission (SRTM; shown at left). The resulting data is then mapped using the image() function in R.
Posted in GDAL, R Data Import, R Graphics, Spatial Analysis
Comments Off on Crop Raster Images in R