Composed or nested queries returning tuples can result in client-side filtering #40
Description
Opened at codeplex by: latkin
When F# queries which return tuples are composed or nested, the queries are not translated efficiently, resulting in client-side filtering where server-side filtering should have been possible.
This becomes very problematic in big-data scenarios: you test with a small data set and results look correct, but when you move to big data set you are unintentionally sucking down tons of data to the client when you thought the query resulted in server-side filtering.
Issue is related to the gymnastics done to swap in mutable tuples required during query translation, then switching back to immutable tuples on the output end.
Repro:
#r "FSharp.Data.TypeProviders"
#r "System.Data.Entity"
#r "System.Data.Services.Client"
open Microsoft.FSharp.Data.TypeProviders
type Northwind = ODataService< "http://services.odata.org/V2/Northwind/Northwind.svc/" >
let client = Northwind.GetDataContext()
client.DataContext.SendingRequest.Add (fun x -> printfn "requesting %A" x.Request.RequestUri)Proper server-side filtering done here:
requesting http://services.odata.org/V2/Northwind/Northwind.svc/Customers()?$filter=substringof('S',ContactName)& $top=10 &$select=*
'''fsharp
let r =
query { for g in client.Customers do
where (g.ContactName.Contains "S")
take 10
select (g, 1) }
for (g, _) in r do
printfn "Name %s" g.ContactName
Improper client-side filtering done here when query is composed:
requesting http://services.odata.org/V2/Northwind/Northwind.svc/Customers()?$filter=substringof('S',ContactName)&$select=*
```fsharp
let r1 =
let a =
query
{ for g in client.Customers do
where (g.ContactName.Contains "S")
select (g, 1) }
let b =
query { for g in a do take 10}
b
for (g, _) in r1 do
printfn "Name %s" g.ContactName
Similarly with nested queries:
let r2 =
let b =
query {
for g in
(query {
for g in client.Customers do
where (g.ContactName.Contains "S")
select (g, 1) }
) do
take 10 }
b
for (g, _) in r2 do
printfn "Name %s" g.ContactName