Logo Deedle

Delay-loaded series

The DelayedSeries type provides an efficient way to create series whose data is loaded on-demand. For example, you may have a large time series stored in a CSV file or in a database and you do not want to load all the data in memory if the user only needs a small part of it.

When you create a delayed series, you specify the overall range of the series (i.e. the minimum and maximum key value) and you provide a function that loads a specified sub-range of the series. When the user accesses a continuous range of the series, the loading function is called to retrieve the data.

Creating a delayed series

To create a delayed series, we need a function that generates data for a given range. The following function generates a series with random data for a given date range with a day frequency:

let generate (low:DateTime) (high:DateTime) : seq<KeyValuePair<DateTime,float>> = 
    let rnd = Random()
    let days = int (high - low).TotalDays
    seq [ for d in 0 .. days -> KeyValuePair(low.AddDays(float d), rnd.NextDouble()) ]

Now we use DelayedSeries.FromValueLoader to create a delayed series. It takes the overall minimum and maximum key of the series and a function that loads data for a sub-range. The loading function gets the lower and upper bound as a tuple of (key, BoundaryBehavior) values where BoundaryBehavior is either Inclusive or Exclusive:

let min = DateTime(2010, 1, 1)
let max = DateTime(2013, 1, 1)

let ls = DelayedSeries.FromValueLoader(min, max, fun (lo, lob) (hi, hib) -> async {
    printfn "Query: %A - %A" lo hi
    let lo = if lob = BoundaryBehavior.Inclusive then lo else lo.AddDays(1.0)
    let hi = if hib = BoundaryBehavior.Inclusive then hi else hi.AddDays(-1.0)
    return generate lo hi })

The key thing about the above is that, so far, no data has been loaded. The loading function is called only when we access part of the series.

Slicing and using delayed series

We can now use the series as usual - for example, to get data for the entire year 2012:

let slice = ls.[DateTime(2012, 1, 1) .. DateTime(2012, 12, 31)]
slice
val slice: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012]) 

val it: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012])

Similarly, we can add the delayed series to a data frame. When doing this, Deedle will only load the data that is needed. In the following example, we add the series to a frame and then access only a slice:

let df = frame ["Values" => ls]
let slicedDf = df.Rows.[DateTime(2012,6,1) .. DateTime(2012,6,30)]
slicedDf
Query: 01/01/2010 00:00:00 - 01/01/2013 00:00:00
Query: 06/01/2012 00:00:00 - 06/30/2012 00:00:00
val df: Frame<DateTime,string> =
  
              Values               
01/01/2010 -> 0.8641869772575418   
01/02/2010 -> 0.025492407318217447 
01/03/2010 -> 0.05260742304386934  
01/04/2010 -> 0.7652885743145077   
01/05/2010 -> 0.04055628339625428  
01/06/2010 -> 0.38164654745997884  
01/07/2010 -> 0.36119195494801304  
01/08/2010 -> 0.33970938927154537  
01/09/2010 -> 0.30610420245023084  
01/10/2010 -> 0.7505037314151409   
01/11/2010 -> 0.21104041680795715  
01/12/2010 -> 0.4199434004966509   
01/13/2010 -> 0.013232222847105457 
01/14/2010 -> 0.3592529471009902   
01/15/2010 -> 0.7683059764699933   
:             ...                  
12/18/2012 -> 0.940391762511769    
12/19/2012 -> 0.14647579755931295  
12/20/2012 -> 0.7690208394523302   
12/21/2012 -> 0.38694795399195125  
12/22/2012 -> 0.8686611222034579   
12/23/2012 -> 0.9989751510486637   
12/24/2012 -> 0.3161712400465776   
12/25/2012 -> 0.6917360330066885   
12/26/2012 -> 0.38302267660753564  
12/27/2012 -> 0.15328827943209777  
12/28/2012 -> 0.10024847440723017  
12/29/2012 -> 0.9184941418269585   
12/30/2012 -> 0.7870134745646498   
12/31/2012 -> 0.22186175086335158  
01/01/2013 -> 0.536576987093919    

val slicedDf: Frame<DateTime,string> =
  
              Values               
06/01/2012 -> 0.06318392794669181  
06/02/2012 -> 0.33073679100490383  
06/03/2012 -> 0.9016609745463129   
06/04/2012 -> 0.36366330191068685  
06/05/2012 -> 0.536593899216576    
06/06/2012 -> 0.6552345067988014   
06/07/2012 -> 0.12942937658722486  
06/08/2012 -> 0.3061267870766172   
06/09/2012 -> 0.3944146818740827   
06/10/2012 -> 0.05544007514637517  
06/11/2012 -> 0.658734228720279    
06/12/2012 -> 0.518301798070686    
06/13/2012 -> 0.187459071477028    
06/14/2012 -> 0.6792054586403198   
06/15/2012 -> 0.30902304412749393  
06/16/2012 -> 0.009193968657296936 
06/17/2012 -> 0.48943915404095517  
06/18/2012 -> 0.08947308585209779  
06/19/2012 -> 0.6829024793172848   
06/20/2012 -> 0.5934482796331055   
06/21/2012 -> 0.41536574112592795  
06/22/2012 -> 0.14302585205789897  
06/23/2012 -> 0.9218096204146516   
06/24/2012 -> 0.8106447709509779   
06/25/2012 -> 0.6866391782111545   
06/26/2012 -> 0.044457230513345825 
06/27/2012 -> 0.6012926875682479   
06/28/2012 -> 0.6538463111888093   
06/29/2012 -> 0.17597400364221483  
06/30/2012 -> 0.853748558832609    

val it: Frame<DateTime,string> =
  
              Values               
06/01/2012 -> 0.06318392794669181  
06/02/2012 -> 0.33073679100490383  
06/03/2012 -> 0.9016609745463129   
06/04/2012 -> 0.36366330191068685  
06/05/2012 -> 0.536593899216576    
06/06/2012 -> 0.6552345067988014   
06/07/2012 -> 0.12942937658722486  
06/08/2012 -> 0.3061267870766172   
06/09/2012 -> 0.3944146818740827   
06/10/2012 -> 0.05544007514637517  
06/11/2012 -> 0.658734228720279    
06/12/2012 -> 0.518301798070686    
06/13/2012 -> 0.187459071477028    
06/14/2012 -> 0.6792054586403198   
06/15/2012 -> 0.30902304412749393  
06/16/2012 -> 0.009193968657296936 
06/17/2012 -> 0.48943915404095517  
06/18/2012 -> 0.08947308585209779  
06/19/2012 -> 0.6829024793172848   
06/20/2012 -> 0.5934482796331055   
06/21/2012 -> 0.41536574112592795  
06/22/2012 -> 0.14302585205789897  
06/23/2012 -> 0.9218096204146516   
06/24/2012 -> 0.8106447709509779   
06/25/2012 -> 0.6866391782111545   
06/26/2012 -> 0.044457230513345825 
06/27/2012 -> 0.6012926875682479   
06/28/2012 -> 0.6538463111888093   
06/29/2012 -> 0.17597400364221483  
06/30/2012 -> 0.853748558832609
namespace System
namespace System.Collections
namespace System.Collections.Generic
namespace Deedle
namespace Deedle.Indices
val fsi: FSharp.Compiler.Interactive.InteractiveSession
member FSharp.Compiler.Interactive.InteractiveSession.AddPrinter: ('T -> string) -> unit
val o: obj
type obj = Object
val iface: Type
Object.GetType() : Type
val fmt: Reflection.MethodInfo
Type.GetMethod(name: string) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, callConvention: Reflection.CallingConventions, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Reflection.MethodBase.Invoke(obj: obj, parameters: obj array) : obj
Reflection.MethodBase.Invoke(obj: obj, invokeAttr: Reflection.BindingFlags, binder: Reflection.Binder, parameters: obj array, culture: Globalization.CultureInfo) : obj
Multiple items
val string: value: 'T -> string

--------------------
type string = String
val generate: low: DateTime -> high: DateTime -> KeyValuePair<DateTime,float> seq
val low: DateTime
Multiple items
type DateTime = new: date: DateOnly * time: TimeOnly -> unit + 16 overloads member Add: value: TimeSpan -> DateTime member AddDays: value: float -> DateTime member AddHours: value: float -> DateTime member AddMicroseconds: value: float -> DateTime member AddMilliseconds: value: float -> DateTime member AddMinutes: value: float -> DateTime member AddMonths: months: int -> DateTime member AddSeconds: value: float -> DateTime member AddTicks: value: int64 -> DateTime ...
<summary>Represents an instant in time, typically expressed as a date and time of day.</summary>

--------------------
DateTime ()
   (+0 other overloads)
DateTime(ticks: int64) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly) : DateTime
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
val high: DateTime
Multiple items
val seq: sequence: 'T seq -> 'T seq

--------------------
type 'T seq = IEnumerable<'T>
Multiple items
type KeyValuePair = static member Create<'TKey,'TValue> : key: 'TKey * value: 'TValue -> KeyValuePair<'TKey,'TValue>
<summary>Creates instances of the <see cref="T:System.Collections.Generic.KeyValuePair`2" /> struct.</summary>

--------------------
type KeyValuePair<'TKey,'TValue> = new: key: 'TKey * value: 'TValue -> unit member Deconstruct: key: byref<'TKey> * value: byref<'TValue> -> unit member ToString: unit -> string member Key: 'TKey member Value: 'TValue
<summary>Defines a key/value pair that can be set or retrieved.</summary>
<typeparam name="TKey">The type of the key.</typeparam>
<typeparam name="TValue">The type of the value.</typeparam>


--------------------
KeyValuePair ()
KeyValuePair(key: 'TKey, value: 'TValue) : KeyValuePair<'TKey,'TValue>
Multiple items
val float: value: 'T -> float (requires member op_Explicit)

--------------------
type float = Double

--------------------
type float<'Measure> = float
val rnd: Random
Multiple items
type Random = new: unit -> unit + 1 overload member GetHexString: stringLength: int * ?lowercase: bool -> string + 1 overload member GetItems<'T> : choices: ReadOnlySpan<'T> * length: int -> 'T array + 2 overloads member GetString: choices: ReadOnlySpan<char> * length: int -> string member Next: unit -> int + 2 overloads member NextBytes: buffer: byte array -> unit + 1 overload member NextDouble: unit -> float member NextInt64: unit -> int64 + 2 overloads member NextSingle: unit -> float32 member Shuffle<'T> : values: Span<'T> -> unit + 1 overload ...
<summary>Represents a pseudo-random number generator, which is an algorithm that produces a sequence of numbers that meet certain statistical requirements for randomness.</summary>

--------------------
Random() : Random
Random(Seed: int) : Random
val days: int
Multiple items
val int: value: 'T -> int (requires member op_Explicit)

--------------------
type int = int32

--------------------
type int<'Measure> = int
val d: int
DateTime.AddDays(value: float) : DateTime
Random.NextDouble() : float
val min: DateTime
val max: DateTime
val ls: Series<DateTime,float>
type DelayedSeries = static member FromIndexVectorLoader: scheme: IAddressingScheme * vectorBuilder: IVectorBuilder * indexBuilder: IIndexBuilder * min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<IIndex<'K> * IVector<'V>>> -> Series<'K,'V> (requires equality) + 1 overload static member FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison) + 1 overload
<summary> This type exposes a single static method `DelayedSeries.Create` that can be used for constructing data series (of type <c>Series&lt;K, V&gt;</c>) with lazily loaded data. You can use this functionality to create series that represents e.g. an entire price history in a database, but only loads data that are actually needed. For more information see the [lazy data loading tutorial](../lazysource.html). </summary>
<example> Assuming we have a function <c>generate lo hi</c> that generates data in the specified <c>DateTime</c> range, we can create lazy series as follows: <code> let ls = DelayedSeries.Create(min, max, fun (lo, lob) (hi, hib) -&gt; async { printfn "Query: %A - %A" (lo, lob) (hi, hib) return generate lo hi }) </code> The arguments <c>min</c> and <c>max</c> specify the complete range of the series. The function passed to <c>Create</c> is called with minimal and maximal required key (<c>lo</c> and <c>hi</c>) and with two values that specify boundary behaviour. </example>
<category>Specialized frame and series types</category>
static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: ('K * BoundaryBehavior -> 'K * BoundaryBehavior -> Async<KeyValuePair<'K,'V> seq>) -> Series<'K,'V> (requires comparison)
static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Threading.Tasks.Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison)
val lo: DateTime
val lob: BoundaryBehavior
val hi: DateTime
val hib: BoundaryBehavior
val async: AsyncBuilder
val printfn: format: Printf.TextWriterFormat<'T> -> 'T
type BoundaryBehavior = | Inclusive | Exclusive
<summary> Specifies the boundary behavior for the `IIndexBuilder.GetRange` operation (whether the boundary elements should be included or not) </summary>
union case BoundaryBehavior.Inclusive: BoundaryBehavior
val slice: Series<DateTime,float>
val df: Frame<DateTime,string>
val frame: columns: ('a * #ISeries<'c>) seq -> Frame<'c,'a> (requires equality and equality)
<summary> A function for constructing data frame from a sequence of name - column pairs. This provides a nicer syntactic sugar for `Frame.ofColumns`. </summary>
<example> To create a simple frame with two columns, you can write: <code> frame [ "A" =&gt; series [ 1 =&gt; 30.0; 2 =&gt; 35.0 ] "B" =&gt; series [ 1 =&gt; 30.0; 3 =&gt; 40.0 ] ] </code></example>
<category>Frame construction</category>
val slicedDf: Frame<DateTime,string>
property Frame.Rows: RowSeries<DateTime,string> with get
<category>Accessors and slicing</category>

Type something to start searching.