File access trouble

Post Reply
belchingmatt
Posts: 2893
Joined: Tue Jan 31, 2006 12:00 am
Location: Spartaaa
File access trouble

Post by belchingmatt »

Right we've had the network guy come around and he is going to give us a quote for an upgrade to cat6 for the technical team. The only trouble with that is that we don't know if it will fix the problem. A simple test would be to buy 50m of cat6 and run it around the floor to see if that solves it, but I can't for the life of me find any reputable sellers online.



Any suggestions anyone as Dabs, RS and Maplin only provide the required length in bulk and I can't be arsed to make my own?





Specs:



Cat5e cables for everything we have.

Don't know for the building but it was probably done 8 years ago.

2 Gigabit switches

All relevant workstations and servers are gigabit.

NAS box has 2 gigabit LAN ports, although I'm not sure if the second one does anything yet but tt does have a flashing LED though. <!--emo&:P--><img src='http://i2.ifrm.com/html/emoticons/tongue.gif' border='0' style='vertical-align:middle' alt='tongue.gif' /><!--endemo-->



Trouble:



50GB files on the NAS can't be accessed by the workstations so they have to be used locally. The files can be moved across the network from the NAS to the workstation where they do work. The workstations only read part of the file in normal use.

As far as I know smaller files such as 1 or 2 gig do work but the larger ones do not.

The app which is trying to use the file crashes out when this happens.



The first thing I tried was to put the 50gig file on a server and the result was that the app didn't crash this time, but it certainly didn't work.



Next thing was to run the app on a laptop that was connected directly to the switch, therefore bypassing the old and dubious office cabling. At this point the app does work correctly. However this was only tried on a smaller file and I need to test the 50gig one later today. The other trouble there is that people are using the machines that I need to use.



I'm thinking now that I'll just buy 50m cat5e and see if that works, at least that would eliminate or not whether it is an office cable legacy.





Any thoughts, suggestions or questions?








--------------------


<div class='signature'> I'm not addicted to porn, I'm addicted to masturbation. Porn is just an enabler.



<img src='http://i3.photobucket.com/albums/y81/ma ... Threat.gif' border='0' alt='user posted image'>

There are plenty more fish in the sea, get your tackle out and see what bites.



Divers go deeper and stay down for longer.
belchingmatt
Posts: 2893
Joined: Tue Jan 31, 2006 12:00 am
Location: Spartaaa
File access trouble

Post by belchingmatt »

Right I've managed to find about 30m cat5e cable of dubious quality and that is now run straight from the switch to a workstation. We have a connection of about 100Mps and getting an average of 40% utilisation. The app is non-responsive so it looks like it is trying. But sadly it appears to be taking too long.




--------------------


<div class='signature'> I'm not addicted to porn, I'm addicted to masturbation. Porn is just an enabler.



<img src='http://i3.photobucket.com/albums/y81/ma ... Threat.gif' border='0' alt='user posted image'>

There are plenty more fish in the sea, get your tackle out and see what bites.



Divers go deeper and stay down for longer.
JJW009
Site Admin
Posts: 6538
Joined: Thu Jul 14, 2005 12:00 am
Location: Midlands
Contact:
File access trouble

Post by JJW009 »

belchingmatt wrote:However this was only tried on a smaller file and I need to test the 50gig one later today.
Until you've tried this test, there is no way to advise.



It depends on the application, but I can give a similar example. Certain video files have the crucial indexing at the end of the file, and the media player has to scan the whole damned file to locate it. If you opened a 50GB badly-indexed divx file over a 100MB network connection, it would not open for almost 3 hours. Over a 1GB connection, it would open in 17 minutes. A 2GB file on a 1Gb connection would open in just 30 seconds.



In other words, it could take just as long to open the file as it does to copy it, depending on how the application scans the file to locate the parts it needs first. The more intelligent the scanning the faster it will open, but it will always take 10x longer on the slower connection.



If the file can be re-indexed, then that can make a huge difference. The application can then go directly to the correct part, rather than scanning for it. This is something I have to do fairly regularly on one of our applications when the database gets it's knickers in a twist. After re-indexing, creating a report can be thousands of times faster.




--------------------


<div class='signature'> <img src='http://myanimelist.net/signature/JJW009.png' border='0' alt='user posted image'> ??? <img src='http://imagegen.last.fm/LastfmMyspace/oartists/5/JJW009.gif' border='0' alt='user posted image'>
belchingmatt
Posts: 2893
Joined: Tue Jan 31, 2006 12:00 am
Location: Spartaaa
File access trouble

Post by belchingmatt »

Hmm that makes a lot of sense. With the money we have to pay for licenses for the app I would damn well hope that the files are well indexed. Anyway we know of other companies with similar setups who have no issue at all.



Also the files work perfectly well when running from an external drive over USB2, and as that is only a couple of hundred megabits it shows that it is currently faster than the cable I tried could provide, but much slower than a gigabit network.



I've got the guys coming back in. They are making me up some cat6 cables to run between the NAS, switch and workstation, representing what a full cable installation would be like. If that doesn't work I don't know what will, fibre?



The daft thing is that this building will be demolished in about 3 years so we don't want to blow loads of cash on something that will last into the next century.

<!--emo&:rant:--><img src='http://i2.ifrm.com/4013/140/emo/angry1.gif' border='0' style='vertical-align:middle' alt='angry1.gif' /><!--endemo-->




--------------------


<div class='signature'> I'm not addicted to porn, I'm addicted to masturbation. Porn is just an enabler.



<img src='http://i3.photobucket.com/albums/y81/ma ... Threat.gif' border='0' alt='user posted image'>

There are plenty more fish in the sea, get your tackle out and see what bites.



Divers go deeper and stay down for longer.
JJW009
Site Admin
Posts: 6538
Joined: Thu Jul 14, 2005 12:00 am
Location: Midlands
Contact:
File access trouble

Post by JJW009 »

It could also be that when accessing the file over USB or locally, you can actually access the file by block number. You can do this to a limit of 90GB or so, depending on the OS. Over a network it's more complicated. Access upto 2GB is simple. Upto 4GB is usually not a problem, but over 4GB file access methods are limited. If the application is doing non-sequential reads it may crash.



This is one reason why with large databases, you would usually have a database server such as SQL. The application doesn't see the huge data file - it passes the request to the server which then returns the correct data in suitably small chunks.



You will find out when you try the large file on the laptop later...



You may need to work around the problem. For example, install the software on the server with the database and access it remotely using terminal server.




--------------------


<div class='signature'> <img src='http://myanimelist.net/signature/JJW009.png' border='0' alt='user posted image'> ??? <img src='http://imagegen.last.fm/LastfmMyspace/oartists/5/JJW009.gif' border='0' alt='user posted image'>
belchingmatt
Posts: 2893
Joined: Tue Jan 31, 2006 12:00 am
Location: Spartaaa
File access trouble

Post by belchingmatt »

How do you know all this?



I don't think I'll try with the laptop. I'll just let the guys come in and test when they've made the cable up. The app company have assured us that everything should work so it looks likely that it is a problem with our infrastructure. Another company says we should be getting a server and we'll at least be able to test that option when they come in again.



We'll probably have to get a sever and cabling done, and at that point I'll need to get better aircon for the comms room. <!--emo&:rofl:--><img src='http://i2.ifrm.com/4013/140/emo/rofl.gif' border='0' style='vertical-align:middle' alt='rofl.gif' /><!--endemo-->









I should read more articles and spend less time on /b/ and looking at porn.




--------------------


<div class='signature'> I'm not addicted to porn, I'm addicted to masturbation. Porn is just an enabler.



<img src='http://i3.photobucket.com/albums/y81/ma ... Threat.gif' border='0' alt='user posted image'>

There are plenty more fish in the sea, get your tackle out and see what bites.



Divers go deeper and stay down for longer.
belchingmatt
Posts: 2893
Joined: Tue Jan 31, 2006 12:00 am
Location: Spartaaa
File access trouble

Post by belchingmatt »

I think that sounded bad so I'll elaborate.



I don't think I'll try with the laptop because it's a pain in the fluffy kitten as it will screw up the network license server and I'll need to redirect manually. Had it been last week when almost everyone was out of the office it wouldn't have been an issue.



<!--emo&:D--><img src='http://i2.ifrm.com/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif' /><!--endemo-->




--------------------


<div class='signature'> I'm not addicted to porn, I'm addicted to masturbation. Porn is just an enabler.



<img src='http://i3.photobucket.com/albums/y81/ma ... Threat.gif' border='0' alt='user posted image'>

There are plenty more fish in the sea, get your tackle out and see what bites.



Divers go deeper and stay down for longer.
User avatar
RedeyesUK
Posts: 2598
Joined: Sun Jul 17, 2005 12:00 am
Location: Over there, then left a bit..no, a bit more..that's it!
File access trouble

Post by RedeyesUK »

In spite of the problems, I'm jealous of your network. You should try ours!



Even our LAN runs at 10Mbps, and I now have to access files on another site, just over the road. However, the way the WAN is setup means that the file I'm trying to access has to come from the site over the road, go all the way up to Scotland, through the head office systems, and then back down to me again!

This means that copying an 8Mb spreadsheet takes about 10 minutes. <!--emo&<_<--><img src='http://i2.ifrm.com/html/emoticons/dry.gif' border='0' style='vertical-align:middle' alt='dry.gif' /><!--endemo-->



However, unlike your company, mine has no plans whatsoever to upgrade in the near future. I'll just have to put up with it! <!--emo&:rant:--><img src='http://i2.ifrm.com/4013/140/emo/angry1.gif' border='0' style='vertical-align:middle' alt='angry1.gif' /><!--endemo-->



What JJ was saying about indexing makes sense, as trying to open that 8Mb file directly in excel will leave excel hanging for about 1/2 hour, then it crashes. <!--emo&:whip:--><img src='http://i2.ifrm.com/4013/140/emo/whip.gif' border='0' style='vertical-align:middle' alt='whip.gif' /><!--endemo-->



I shall be watching this thread with interest!



Any idea if it's possible to get some sort of WiFi link to the other site? It's about 100m away, and there is line of sight (more or less, depending on trees!).
JJW009
Site Admin
Posts: 6538
Joined: Thu Jul 14, 2005 12:00 am
Location: Midlands
Contact:
File access trouble

Post by JJW009 »

RedeyesUK wrote:Any idea if it's possible to get some sort of WiFi link to the other site? It's about 100m away, and there is line of sight (more or less, depending on trees!).
It's a lot harder than you'd think. Sticking a couple of cheap USB adaptors up a pole will seam to work, but will tend to be unreliable. You really need good (expensive) directional aerials. It's also a good idea to use 802.11a because it uses a different feqencyur to 802.11b ro g and so gets less interference.



Also trees are a ealr roblem!p



wt? Fiefox iscrmbling m let S:sretya srf






--------------------


<div class='signature'> <img src='http://myanimelist.net/signature/JJW009.png' border='0' alt='user posted image'> ??? <img src='http://imagegen.last.fm/LastfmMyspace/oartists/5/JJW009.gif' border='0' alt='user posted image'>
User avatar
RedeyesUK
Posts: 2598
Joined: Sun Jul 17, 2005 12:00 am
Location: Over there, then left a bit..no, a bit more..that's it!
File access trouble

Post by RedeyesUK »

JJW009 wrote:wt? Fiefox iscrmbling m let S:sretya srf
<!--emo&:lol:--><img src='http://i2.ifrm.com/html/emoticons/laugh.gif' border='0' style='vertical-align:middle' alt='laugh.gif' /><!--endemo-->



Sure it's Firefox doing that and not booze? <!--emo&:P--><img src='http://i2.ifrm.com/html/emoticons/tongue.gif' border='0' style='vertical-align:middle' alt='tongue.gif' /><!--endemo-->





I know what you mean about the wifi, I was never under the impression that cheap ones would work (If I thought that, I'd have tried it ages ago!) I was thinking along the lines of directional aeriels, like you suggested. I just wasn't sure if that was technically feasible.



I might make a few phone calls next week, see if I can get some prices. <!--emo&:)--><img src='http://i2.ifrm.com/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif' /><!--endemo-->
JJW009
Site Admin
Posts: 6538
Joined: Thu Jul 14, 2005 12:00 am
Location: Midlands
Contact:
File access trouble

Post by JJW009 »

DIY aerials can be very good - Cantena are well documented, but finding the "sweet spot" takes a lot of attention to detail. You have to align things by a fraction of a millimeter! The easiest DIY you can do is with an old Squarial or other satellite dish, and you can get a range of miles with that.




--------------------


<div class='signature'> <img src='http://myanimelist.net/signature/JJW009.png' border='0' alt='user posted image'> ??? <img src='http://imagegen.last.fm/LastfmMyspace/oartists/5/JJW009.gif' border='0' alt='user posted image'>
belchingmatt
Posts: 2893
Joined: Tue Jan 31, 2006 12:00 am
Location: Spartaaa
File access trouble

Post by belchingmatt »

Problem resolved. We don't need a new server or cable installation. <!--emo&:o--><img src='http://i2.ifrm.com/html/emoticons/ohmy.gif' border='0' style='vertical-align:middle' alt='ohmy.gif' /><!--endemo-->



In fact it would appear JJ hit the nail on head earlier on with a mention of the index. Anyway when I first saw the problem I was just working with the data files, and it was only today when we got someone in to do some better testing with cat6 cables that I saw the data files in another location with corresponding index files.



Normally the software can create the index file from only the data file quite quickly, but as we are talking monster files that process is obviously quite lengthy, and so when we give up on the hung application after 60 mins it probably hasn't even done half of the index.



In this case it is important to use the index file that is supplied with the data. Something the main technical guy in the UK told us we didn't need to do. When we called someone more junior they were able to confirm that they are required.



It's not the first time and certainly won't be the last where this sort of issue arises as a result of poor documentation and no overlap between users and admin.








--------------------


<div class='signature'> I'm not addicted to porn, I'm addicted to masturbation. Porn is just an enabler.



<img src='http://i3.photobucket.com/albums/y81/ma ... Threat.gif' border='0' alt='user posted image'>

There are plenty more fish in the sea, get your tackle out and see what bites.



Divers go deeper and stay down for longer.

Post Reply