Because at the moment I can't syndicate my comments through blogger, I thought I would promte a comment from Mike Champion (The program manager for XLinq) to the front page of my blog. So other people get the chance to see this issue more publically and so you (the reader) can respond too if you so wish.
I will be trying to reply to the comment properly by the end of the week, I am on vacation at the moment so I will try and do some thinking when I get chance :) But I will respond properly.
Anyway, here is the comment.
Hi, I'm the program manager for XLinq at Microsoft. I wanted to let you know that we are looking into this very problem right now. It would be good to hear from you and others in more detail about how your big XML file is structured. Your idea of having a LINQ-queryable XmlReader stream is one we have considered, but that doesn't really leverage the rest of XLinq. We'd prefer something akin to the XStreamingElement class in the May CTP, where a repeated element structure is evaluated "lazily". The trick is to define the structure of the streaming input without a) requiring a schema, b) requiring the user to learn a different technology such as XPath (remember that XLinq is not necessarily aimed at an audience of XML experts who already know such things), and c) making it so complex that users might as well use XmlReader to do the job. Specific question: does your 900MB document have a regular structure, e.g. is it 900,000 1K elements that have the same structure, 900 1MB documents with varying structures, one big amorphous thing, or what? Is there some less structured "header" information at the beginning before any regular repeating structure begins? I think we'll be able to offer something that is simple to use and powerful for the case where large documents consist of many relatively well-structured top-level elements, but we're wondering how much complexity beyond that we can feasibly support before saying "just use XmlReader".Thanks! You can contact me via the "contact us" form at blogs.msdn.com/mikechampion if you want to follow up, or leave a comment in one of the entries there.
