Let me start this post by stating that I am not even sure if what I am trying to do is legal here. But from reading the docs, it does appear to be a valid use of the API, and it does work, most of the time.

The full code can be found here: https://gist.github.com/ayende/7495987

The gist of it is that I am trying to do two things:

• Write to a file opened with FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING.
• Read from this file using a memory map.
• Occasionally, I get into situations where after I wrote to the file, I am not reading what I wrote.

I have a repro, and we reproduced this on multiple machines. Both Windows 7 and Windows 8.

Here is the relevant code (the full code is in the link), explanation on it below:

   1: const uint nNumberOfBytesToWrite = 4096*3;

   2: var buffer = (byte*)(VirtualAlloc(IntPtr.Zero, new UIntPtr(nNumberOfBytesToWrite), AllocationType.COMMIT, MemoryProtection.READWRITE)

   3:             .ToPointer());

   4:  

   5: for (int i = 0; i < nNumberOfBytesToWrite; i++)

   6: {

   7:     *(buffer + i) = 137;

   8: }

   9:  

  10: var g = Guid.NewGuid().ToString();

  11:  

  12: var safeHandle = CreateFile(g,

  13:     NativeFileAccess.GenericRead | NativeFileAccess.GenericWrite,

  14:     NativeFileShare.Read, IntPtr.Zero,

  15:     NativeFileCreationDisposition.OpenAlways,

  16:     NativeFileAttributes.Write_Through | NativeFileAttributes.NoBuffering | NativeFileAttributes.DeleteOnClose,

  17:     IntPtr.Zero);

  18:  

  19: var fileStream = new FileStream(safeHandle, FileAccess.ReadWrite);

  20: fileStream.SetLength(1024 * 1024 * 1024); // 1gb

  21:  

  22: if (safeHandle.IsInvalid)

  23: {

  24:     throw new Win32Exception();

  25: }

  26:  

  27: FileStream mms = fileStream;

  28: //mms = new FileStream(g, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete);

  29: var mmf = MemoryMappedFile.CreateFromFile(mms, Guid.NewGuid().ToString(), fileStream.Length,

  30:     MemoryMappedFileAccess.Read, null, HandleInheritability.None, true);

  31:  

  32: MemoryMappedViewAccessor accessor = mmf.CreateViewAccessor(0, fileStream.Length, MemoryMappedFileAccess.Read);

  33: byte* ptr = null;

  34: accessor.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);

  35:  

  36: Task.Factory.StartNew(() =>

  37: {

  38:     long lastPos = 0;

  39:     while (true)

  40:     {

  41:         int count = 0;

  42:         while (true)

  43:         {

  44:             if (*(ptr + lastPos) != 137)

  45:             {

  46:                 break;

  47:             }

  48:             lastPos += 4096;

  49:             count ++;

  50:         }

  51:         Console.WriteLine();

  52:         Console.WriteLine("Verified {0} MB", count * 4 / 1024);

  53:         Console.WriteLine();

  54:         Thread.Sleep(2000);

  55:     }

  56: });

  57:  

  58: for (int i = 0; i < 1024*64; i++)

  59: {

  60:     var pos = i*nNumberOfBytesToWrite;

  61:     if (i%100 == 0)

  62:         Console.Write("\r{0,10:#,#} kb", pos/1024);

  63:     var nativeOverlapped = new NativeOverlapped

  64:     {

  65:         OffsetHigh = 0,

  66:         OffsetLow = (int) pos

  67:     };

  68:  

  69:     uint written;

  70:     if (WriteFile(safeHandle, buffer, nNumberOfBytesToWrite, out written, &nativeOverlapped) == false)

  71:         throw new Win32Exception();

  72:  

  73:     for (int j = 0; j < 3; j++)

  74:     {

  75:         if (*(ptr + pos) != 137)

  76:         {

  77:             throw new Exception("WTF?!");

  78:         }

  79:         pos += 4096;

  80:     }

  81: }

This code is doing the following:

• We setup a file handle using NoBuffering | Write_Through, and we also map the file using memory map.
• We write 3 pages (12Kb) at a time to the file.
• After the write, we are using memory map to verify that we actually wrote what we wanted to the file.
• _At the same time_ we are reading from the same memory in another thread.
• Occasionally, we get an error where the data we just wrote to the file cannot be read back.

Now, here is what I think is actually happening:

• When we do an unbuffered write, Windows has to mark the relevant pages as invalid.
• I _think_ that it does so before it actually perform the write.
• If you have another thread that access that particular range of memory at the same time, it can load the _previously_ written data.
• The WriteFile actually perform the write, but the pages that map to that portion of the file have already been marked as loaded.
• At that point, when we use the memory mapped pointer to access the data, we get the data that was there before the write.

As I said, the code above can reproduce this issue (you might have to run it multiple times).

I am not sure if this is something that is valid issue or just me misusing the code. The docs are pretty clear about using regular i/o & memory mapped i/o. The OS is responsible to keeping them coherent with respect to one another. However, that is certainly not the case here.

It might be that I am using a single handle for both, and Windows does less checking when that happens? For what it is worth, I have also tried it using different handles, and I don’t see the problem in the code above, but I have a more complex scenario where I do see the same issue.

Of course, FILE_FLAG_OVERLAPPED is not specified, so what I would actually expect here is serialization of the I/O, according to the docs. But mostly I need a sanity check to tell me if I am crazy.