Back to Questions
2.4k
views1
comment
Pairwise Sequence Processing to compare db tables
f#parallel

Use case:

Iterate through 2 db tables in parallel and find differences and gaps/missing records in either table. Assume that 1) pk of table is an Int ID field; 2) the tables are read in ID order; 3) records may be missing from either table (with corresponding sequence gaps).

I'd like to do this in a single pass over each db - using lazy reads. My initial thought was to use pairwise sequence processing and use Seq.skip to try and keep the table processing in sync. However apparently this is very slow as Seq.skip has a high overhead (creating new sequences under the hood) so this could be a problem with a large table (say 200k recs).

This is an interesting problem and one of my first forays into the fp world - so I'd like to get some insights on the most efficient approach here.

Anyone care to comment?

TIA

.

It's not a functional way, but I'd suggested you to use IEnumerator instead of IEnumerable.

1
2
3
4
5
6
7
8

let enum1, enum2 = seq1.GetEnumerator(), seq2.GetEnumerator()
let nempty1, nempty2 = ref enum1.MoveNext(), ref enum2.MoveNext()
while !nempty1 && !nempty2 do
  if someKindOfEquality enum1.Current enum2.Current then
    nempty1 := enum1.MoveNext()
    nempty2 := enum2.MoveNext()
  else
    // not equal, compare them and move one of enumerators
By on 4/16/2011 3:34 AM ()

    Topic tags

    IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
    Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy     		Built with WebSharper