CodeBetter.Com
CodeBetter.Com
RSS 2.0 via Feedburner
           Do you Twitter? Follow us @CodeBetter

Brendan Tompkins [MVP]

Blog First. Ask Questions Later.

The Story of the Two Dotnet Gnomes and the Network Services Ogre

Once upon a time, there were two dotnet gnomes.  They lived in a cave.  Everyday they got up and slaved away at their computers.  They were creating something that was going to be useful to the world, and they were very excited.  After 10 months of slaving away, eating nuts and berries, they finally launched their beautiful website.  And beautiful it was.  They were very proud.  It was n-tier, scalable, and used every buzz-word they’d heard in the forest, like “XML” and “Web Services” and “MSMQ” and on and on. 

 

When the whole world found out about it, they were all very happy too.  They loved all of the new stuff the two gnomes had built to help them.  Everyone was dancing and singing joyful songs. 

 

Then, one day, the gnomes noticed something funny about their new site.  It seemed slow.  So they looked into the problem.  They were confused because they thought they had done everything right, and couldn’t understand the slowness.  They noticed a very specific problem that they had not heard about before in all their years living in the forest.  Here was the problem:  One very tiny image that was used all over the site was making the pages appear to load very slowly!  So they diligently went about checking the source of this problem. They checked the file.  It was fine.  They checked the HTML, it was fine.  They checked the server resources. They were fine.  They checked the cache hits on IIS.  It was loyally doing what it was supposed to do.  Then they noticed something fishy.

 

When the image was requested from outside the gnome’s local network, it loaded fine!  Better than fine, actually!   It loaded really fast!  They scratched their little gnome heads, and thought that it may possibly be a network issue on their local network.  Well, they aren’t the kind of gnomes that go around point fingers, so they had their little gnome hats in their hands, and knocked on the network services ogre’s door. 

 

“WHAT DO YOU WANT?” asked the ogre. “We noticed a strange problem.” Said the gnomes, and they proceed to tell the ogre the story.  The ogre barked at them “IT’S NOT THE NETWORK!!! YOU GNOMES ARE ALWAYS BLAMING THE NETWORK!!”  Well the gnomes don’t think that way.  They weren’t blaming anyone, they were just trying to get their little site to load fast.  Finally, the ogre yelled at the gnomes “CHECK YOUR SHIT!”  So they went back and “checked their shit,” again.  Their shit was fine.

 

The next day they knocked on the ogre’s door.  He was being particularly ogre-ish on this day, and wouldn’t even consider that his network may be a problem.  He screamed at the gnomes, yelling about how his network has “100 MEGABIT PIPES”, and “PIX” and “SWITCHES” and all kinds of other things that the gnomes didn’t understand too well.   Again, he sent them away, telling them to “CHECK YOUR SHIT!” 

 

The gnomes were really sad at this point. They didn’t understand how they could check their shit any more than they already had.  So they set about, in their gnome-ish way, to try to reason with the ogre.  They pulled out their magic VS.NET bag of tricks, and in 10 minutes, wrote a little windows app that they thought would help prove their point.  It requested this little image over and over, and recorded the time it took to retrieve the image.  Lo and behold, the image loaded over and over again from outside the network at blazing speeds, but from inside the network, the image loaded slowly!  Really slow! We’re talking 6000 times slower at some points.   Same image.  Same server.  So they showed this research to the ogre.  He looked at it and said “THAT DOESN’T PROVE ANYTHING” and after they pleaded with him to consider their argument, he finally walked away saying “I’M NOT WORRIED ABOUT IT, IT’S ONLY INTERNAL ANYWAY.”  

 

So what’s the moral of the story?   Well, the gnomes thought that the good people out there reading this story could help come up with one, because they haven’t a clue.



Comments

Darrell said:

Cynical moral: Network admins are big hairy unfriendly people.

Realistic moral: The ogre is not interested in doing work, as evidenced by the fact that instead of investigating a complaint, he tried to blame anyone else, and then when confronted with evidence, dismissed the issue as unimportant.
# October 31, 2003 2:50 AM

Sebastien Lambla said:

Top of my head, but if the internal route and the external route have different access times (bw aside), the only valid reason for it not working is on the TCP/Ethernet side of things. Few things to think about:
1. Would there be a switch between the server and local that fragment tcp packets?
2. Would there be a bad MTU configuration on one of the ethernet adapters causing an MTU negociation?
3. Is there some kind of IPSec going on the wire to access the server from inside?

Frankly, my best advice, use Ethereal and register the network data for this single request (using your nifty app) and look at the TCP frames directly. If they are the same, look at anything between you and the server.

Just a question, but, is there more latency or slow transfer? And is the image smaller than the TCP frame?
# November 1, 2003 7:44 AM

Brendan Tompkins said:

Sebastien,

Thanks for the tips. I haven't researched the latency vs. transfer, but my guess would be latency is the problem. The image is less than 1K in size. How could I go about veryifying the size of the TCP frame? I'm kinda clueless about this sort of thing.

Thanks again!

Brendan
# November 1, 2003 2:57 PM

Eric said:

Maybe the Ogre would be more agreeable to assisting in this endeavor if the Gnomes promised to buy him a beer?

Seriously,
Does the "slowness" happen on all segements of the internal network or just on a particular segment?

Is the web server in a DMZ? If so, your problem might be with the router between you and the web server.


Eric
# November 3, 2003 10:08 AM

Brendan Tompkins said:

Ha! The Ogre doesn't drink. Perhaps he should... like us Irish gnomes. :)

The Ogre called in a consultant company on Friday who fixed the problem. The problem was due to something called WebSense that the Ogres use to keep gnomes from looking at pictures of naked gnome women at work and downloading any software that could make our jobs easier...

They disabled WebSense for requests from our DMZ. Funny, because when I mentioned WebSense to the Ogre last week, he really barked at me. Anyhow. Our problem is fixed. The Ogre still says that IIS is somehow causing the problem, even though it was fixed by disabling WebSense.
# November 5, 2003 3:13 AM

Brian Desmond said:

This is definetely a convenience of being both the ogre and the gnome - i run the network, fix the network, and write the software ... not much that doesn't get routed through me at some point.
# November 5, 2003 1:29 PM

Sebastien Lambla said:

In my company, I must say we don't really do anything that would get the ogre invovled (even though they are very friendly guys, quite unusual in the IT world. I think I got them happy when they saw my corp laptop had an unapprooved os, with a skin, object dock, a beta of office 2003 and objectdock :) )

Anywy, about the TCP frame problem, what can happen is the following:
Let's say you have a network configuration with this:
A -> B -> C -> D -> E

A: Your web server
B: The IAS Server
C: a network router
D: another network router
E: Your computer

Now, your TCP stream is cut down into IP packets, and they follow a resumable cut / reconstruct algorythm.

For a 1400 bytes image for example, let's say A sends it. The TCP Frame, the packet containing the data, has a payload big enough (1500 to include the IP/TCP header and the data). You got a single packet flowing through the network.
Let's say B filter that packet, and cut it down because C, from time to time, says it has a problem routing the data (this also does happen under high network load). It will try to cut down your single nice packet into 5 packets, with a size of 300 bytes (this is an extreme unprobable case, but I saw it happen once or twice).

Now, your C is a network router, of the Cisco type. It receives the data, and because of priority, sends your packet 1, 2, 3, 4 and 5 in a completely different order (Unprobable but once again, can happen if C is a set of 10 nodes with alternative roads etc).

D may itself be configured to reorder the packets, and then buffers 1, 2, 3, 4 and 5 to get them in the right order. This has a tendendcy of happening if D is another IAS server (because they need to parse the data before sending it).

It will happily send these packets back to your computer, in the correct order. In the mean time, a lot happened.
While you can't do much about reordering, you can analyse the data you receive when you don't have any latency (using Ethereal or netmon) and see the payload of your TCP packet. You can then do the same thing for where you got latency. By seeing how the data differs, you can be pretty sure of a network problem.

Finally, as a tip, if they use WebSense, they use IAS. IAS has a very handy feature which is called the Firewall client, which install on the client machine and goes you through the IAS server without any kind of proxying. And most of the time, if you know the address of your proxy, just try to locate the installation files (there is a documentation on the ms site about where to find it, just look at the IAS doc in msdn).

If you're carefull enough, and if the ogre is not paying attention, you can get rid of any "non porn non hack non development nothing" kind of filters. For free. Oh, yeah, it's very handy for icq, messenger and IRC as well.

Seb
# November 8, 2003 11:47 AM

Alex Tanner said:

Where do I find the IAS docs on MSDN?
# January 16, 2005 7:45 AM

What DOS console commands do I need to send in ord said:

help me please!!!
# January 17, 2005 1:21 AM

Alex Tanner said:

What DOS console commands do I need to type in order to disable Websense?
# January 17, 2005 1:23 AM

anonymous said:

tsshutdn /reboot
# January 17, 2005 2:59 AM

Leave a Comment

(required)  
(optional)
(required)  

Enter the numbers above:
Add

About Brendan Tompkins

Brendan has been programming with .NET since the first public beta and is owner and operator of Port Technology Services, a consultancy company providing .NET application development services to the Maritime industry. In July, 2007, he was awarded the Microsoft MVP award for ASP.NET. He's also a proud co-founder of failed .COM startup Intrinsigo, and has had a hand in the failure of numerous other businesses. He currently runs CodeBetter.Com and Devlicio.us, and lives in Norfolk, Virgina with his wife Tiara and son Ian.

View Brendan's profile on LinkedIn

Check out Devlicio.us!

Our Sponsors

Proudly Partnered With


This Blog

Syndication

News

MVP
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.