Delay-loaded series

The DelayedSeries type provides an efficient way to create series whose data is loaded on-demand. For example, you may have a large time series stored in a CSV file or in a database and you do not want to load all the data in memory if the user only needs a small part of it.

When you create a delayed series, you specify the overall range of the series (i.e. the minimum and maximum key value) and you provide a function that loads a specified sub-range of the series. When the user accesses a continuous range of the series, the loading function is called to retrieve the data.

Creating a delayed series

To create a delayed series, we need a function that generates data for a given range. The following function generates a series with random data for a given date range with a day frequency:

let generate (low:DateTime) (high:DateTime) : seq<KeyValuePair<DateTime,float>> = 
    let rnd = Random()
    let days = int (high - low).TotalDays
    seq [ for d in 0 .. days -> KeyValuePair(low.AddDays(float d), rnd.NextDouble()) ]

Now we use DelayedSeries.FromValueLoader to create a delayed series. It takes the overall minimum and maximum key of the series and a function that loads data for a sub-range. The loading function gets the lower and upper bound as a tuple of (key, BoundaryBehavior) values where BoundaryBehavior is either Inclusive or Exclusive:

let min = DateTime(2010, 1, 1)
let max = DateTime(2013, 1, 1)

let ls = DelayedSeries.FromValueLoader(min, max, fun (lo, lob) (hi, hib) -> async {
    printfn "Query: %A - %A" lo hi
    let lo = if lob = BoundaryBehavior.Inclusive then lo else lo.AddDays(1.0)
    let hi = if hib = BoundaryBehavior.Inclusive then hi else hi.AddDays(-1.0)
    return generate lo hi })

The key thing about the above is that, so far, no data has been loaded. The loading function is called only when we access part of the series.

Slicing and using delayed series

We can now use the series as usual - for example, to get data for the entire year 2012:

let slice = ls.[DateTime(2012, 1, 1) .. DateTime(2012, 12, 31)]
slice

(Delayed series [01/01/2012 .. 12/31/2012]) 

Similarly, we can add the delayed series to a data frame. When doing this, Deedle will only load the data that is needed. In the following example, we add the series to a frame and then access only a slice:

let df = frame ["Values" => ls]
let slicedDf = df.Rows.[DateTime(2012,6,1) .. DateTime(2012,6,30)]
slicedDf

              Values                 
06/01/2012 -> 0.13346853809632686    
06/02/2012 -> 0.5813117473926136     
06/03/2012 -> 0.00041208310028262307 
06/04/2012 -> 0.8286691420016086     
06/05/2012 -> 0.9537808417072333     
06/06/2012 -> 0.4764211325292338     
06/07/2012 -> 0.9538510323929349     
06/08/2012 -> 0.4345892910671896     
06/09/2012 -> 0.5108961426094394     
06/10/2012 -> 0.5230508429752667     
06/11/2012 -> 0.8815297964120345     
06/12/2012 -> 0.3190440702078199     
06/13/2012 -> 0.4863928580755805     
06/14/2012 -> 0.4315127643432025     
06/15/2012 -> 0.47016692158711504    
06/16/2012 -> 0.6309070290650055     
06/17/2012 -> 0.7236910329928876     
06/18/2012 -> 0.17880558563376336    
06/19/2012 -> 0.921307334674331      
06/20/2012 -> 0.2012629241273377     
06/21/2012 -> 0.25050503015789993    
06/22/2012 -> 0.1514585662190877     
06/23/2012 -> 0.09396239725978472    
06/24/2012 -> 0.8547183418077422     
06/25/2012 -> 0.4813356259799012     
06/26/2012 -> 0.7662591568825007     
06/27/2012 -> 0.15516597073792182    
06/28/2012 -> 0.0064519779960432455  
06/29/2012 -> 0.34173511819216673    
06/30/2012 -> 0.2621840078213712