Timeout Property (wodSFTP / wodSFTP.NET / wodSFTPdll)
by kdcarlisle, Thursday, May 28, 2009, 14:42 (5659 days ago)
I have created a powershell cmdlet that does various ftp (get/put/remove) actions (all in blocking mode). This cmdlet is used within our powershell scripts to upload/download files from various ftp servers.
One of the servers that we work on is a titan ftp server. This ftp server is an older version (and is starting to have some issues). We are in the process of migrating it to a new machine, but that will take some time (6-12 months).
Currently the .Connect method hangs at random times. We have various jobs that use this cmdlet to connect to the same ftp server (all using the same ftp account). At some point each of the jobs has gotten hung up while attempting to .Connect. Because of this I set a time-out value in the controlling application (Visual Cron) to stop after 2 hours.
Yesterday I changed the Timeout property (from 0) to 30 right before the .Connect is called. I then set the Timeout propert back to 0 after the .Connect call. I installed the updated code and allowed all the jobs to start processing. This morning when I checked the system various jobs were still timeing out but from the Visual Cron side (not the component side).
While we do have a work-around (using Visual Cron), we would prefer to use the timeout feature of the component (and not have to wait for 2 hours).
-Kent
Re: Timeout Property
by woddrazen, Thursday, May 28, 2009, 15:03 (5659 days ago) @ kdcarlisle
Hi Kent,
How can we help you here? When you set Timeout Property to 30 does it disconnect after 30 seconds?
When you set timeout to 0 it will not autodisconnect at all. Can you explain maybe what action you want to take?
More help for Timeout Property you can find here:
http://www.weonlydo.com/FtpDLX.NET/Help/WeOnlyDo.Client.FTP~WeOnlyDo.Client.FtpDLX~Timeout.html
Regards,
Drazen
Re: Timeout Property
by kdcarlisle, Thursday, May 28, 2009, 15:50 (5659 days ago) @ woddrazen
Hi Drazen,
Originally the code set the Timeout property to 0 (which disables the auto-disconnect). I changed the code to set the value to 30 seconds before the call to .Connect. I was assuming that if the connect had an issue (ie: got hung, server busy, etc.) that it would either (1) return out or (2) throw a TimeoutException.
Once the connection has been made I wanted to set the Timeout property back to a 0 (disabling the auto-disconnect). I did this based on another post in the forum.
After I made those changes and installed the updated code nothing has changed (that I can tell). The occasional hang still happens and is eventually timed out via Visual Cron.
Basically here is what I to happen.
1) cmdlet starts up
2) paramters are validated
3) connection is made to ftp server [if any issue or hang throw exception]
4) cmdlet processes request (get/put/remove)
5) ftp disconnects from server
6) cmdlet shuts down
Since this is all done sequentially we took the simple straight forward approach and used blocking.
I did notice in another post someone mentioned using an undocumented feature .DebugFile. I can add this into the cmdlet if you believe it will write out information before it hangs.
-Kent
Re: Timeout Property
by wodDamir, Thursday, May 28, 2009, 16:32 (5659 days ago) @ kdcarlisle
Hi Kent,
Ok, but is this a *hang* or a simple idling state (since you are preventing auto-disconnection)?
Can you check the states (in State-Changed event) and dump them here?
Do you use events perhaps? Are you doing some processing in them?
Regards,
Damba
Re: Timeout Property
by kdcarlisle, Thursday, May 28, 2009, 19:32 (5659 days ago) @ wodDamir
Hi Damba,
I am assuming it is a *hang*, but I guess it might be an idling state. Can you tell me what will happen if I leave the Timeout property set to 30 seconds? If it is an idle state (rather than a hang) will it throw the TimeoutException?
I am not using any events. It is all straight line code processing the request in order. I can write in some debug statements at various points to see where it stalls.
Let me know what happens when the Timeout value is reached and then I'll make that change and add some debug code.
-Kent
Re: Timeout Property
by wodDamir, Friday, May 29, 2009, 00:07 (5659 days ago) @ kdcarlisle
Kent,
If you leave Timeout property on 30, the component should throw a timeout exception if there is not activity for 30 seconds.
Could you please try finding where exactly does the *hang* occur?
Regards,
Damba
Re: Timeout Property
by kdcarlisle, Friday, May 29, 2009, 20:26 (5658 days ago) @ wodDamir
Hi Damba,
I put in debug code around all the various method calls. Thus far every stall has occurred when called .ListNames. Looking at the debug files I now see that it appears to (ramdomly) do this when it connects to any of our Titan V4 FTP Servers. We also have a Titan V5 FTP Server, but as yet I have not seen this happen when using that machine.
I left the Timeout property set to 30 seconds and the cmdlet now times-out (on those stalls) and exits out.
Any ideas on why it stalls?
-Kent
Re: Timeout Property
by woddrazen, Friday, May 29, 2009, 21:35 (5658 days ago) @ kdcarlisle
Kent,
In order to deal with your problem we will need to duplicate it somehow.
Is there any chance we can connect to one of yours Titan servers and duplicate your issue?
You can send your private informations to techsupport@weonlydo.com
Drazen
Re: Timeout Property
by kdcarlisle, Saturday, June 27, 2009, 20:49 (5629 days ago) @ woddrazen
Damba/Drazen,
I know that we finihsed up this issue in a ticket, but I had a couple more items that recently cropped up that I wanted to pass along.
1) The time-out issue is now happening on the newest version of the Titan FTP Server.
2) Our connections to the FTP server are not passive.
In the last converstion that we had you explained that the issue was when a passive data connection was begin opened. Since then I have re-checked all the jobs (and code) to ensure that they do not use passive connections.
I also use filezilla (3.2.x) on my machines to connect to that same server (active mode) and have never had an issue.
-Kent
Re: Timeout Property
by wodDamir, Saturday, June 27, 2009, 20:53 (5629 days ago) @ kdcarlisle
Kent,
Is this only happening with ListNames? Does ListDir have same effect?
Can you try setting StrictHost property to true before connecting?
Regards,
Damba
Re: Timeout Property
by kdcarlisle, Sunday, June 28, 2009, 22:42 (5628 days ago) @ wodDamir
Damba,
We do not use the ListDir in our code so I am not sure if it too has this same issue. I have updated our code (1) to the lastest version 1.4.5 and (2) set the .StrictHost to true [before connection]. I have updated the server with the new code and will monitor it for the next couple of days. I will post back once I have an answer if the .StrictHost property has resolved the issue.
-Kent
Re: Timeout Property
by kdcarlisle, Monday, July 06, 2009, 15:44 (5620 days ago) @ kdcarlisle
Damba,
Ok I monitored the code for the past week. Using .StrictHost along with version 1.4.5 seems to work. I did not get any time-outs. However, a new problem arose. We had 13 powershell crashes (at various times). Each crash was in the weonlydo.client.ftp and the exception was system.nullreferenceexception. I have reverted back to version 1.4.3 (using .StrictHost) and I continue to get the time-outs. I do have the mini-dump files from those crashes if you would like to see them.
Also, another issue that I found while using the 1.4.5 version seems to be that the .GetFile call is now case sensitive. There was a file (CSFRAG2.txt) on the ftp server that the job could not pick up. It was passing in the name (csfrag2.txt) and it would not find a match. I changed the name to match what was on the server and it downloaded the file. I am re-checking our code to ensure that its not on our side, but the code worked fine (case insensitive) with version 1.4.3.
-Kent
Re: Timeout Property
by woddrazen, Monday, July 06, 2009, 16:08 (5620 days ago) @ kdcarlisle
Kent,
Please send us logs to techsupport@weonlydo.com for crashing issue.
Is there any chance you can send us logs for your other issue? I'm not sure how ti works with 1.4.3 files in GetFile and PutFiles should be case sensitive.
Can you send us some logs so we can check that also?
Drazen
Re: Timeout Property
by kdcarlisle, Monday, July 06, 2009, 21:35 (5620 days ago) @ woddrazen
Drazen,
The 10 mini-dump files that I have are compressed into a 148 meg zip file. I can not email it (have been trying). I can either upload to your ftp (if available) or I can set up an account for you to download it from my local ftp. Let me know how you want to proceed.
As for log files for the other issue, I'm afraid there aren't any. The job tries to download files that match a certain pattern. If there is nothing in the pattern that matches then there is nothing to download. This is why I am a little confused. When I looked at the code it performs a .ListNames to get all the contents of a remote directory. Then it loops through that list and anything that is not a directory (starts with dot or ends with /) gets compared to the pattern. The pattern check code lower cases both values to ensure a match. However, once a match is found, the original file name (from the .ListNames call) is stored in an array to be used when it is time to fetch the file. So, if the server has an upper-case file (CSFRAG2.txt) and the pattern is lower-case (csfrag2.txt) they should actually match and the server version should be used when fetching the file. For some reason this did not work over the weekend. The file remained on the server until I changed the file pattern. I will create a small stand-alone application to further test this.
Re: Timeout Property
by wodDamir, Monday, July 06, 2009, 21:46 (5620 days ago) @ kdcarlisle
Kent,
Can you please try removing anything unneeded from those logs? Only the last few lines before the crash are actually needed, so I don't think 140MB log is necessary.
Could you cut them to few lines that actually show the error (last few commands)?
If that doesn't show anything helpful, we'll provide you access to our FTP server.
Regards,
Damba
Re: Timeout Property
by kdcarlisle, Monday, July 06, 2009, 22:02 (5620 days ago) @ wodDamir
Damba,
These are mini 'Crash Dump Files' that were generated by the OS. When I opened the text file I found the following line.
ErrorSubPath=Genericclr20r3powershell.exe6.1.6585.14819428fweonlydo.client.ftp1.4.5.1484a37bcdb73asystem.nullreferenceexception
If there is something specific that you want me to pull out of the dumps please let me know and I'll see what I can do. I'm not sure what other logging information you want me to pull from.
-Kent
Re: Timeout Property
by kdcarlisle, Monday, July 06, 2009, 22:06 (5620 days ago) @ kdcarlisle
Drazen,
I created a simple standalone app that will test the basics of our powershell snap-in. I have run this code with both versions (1.4.3 and 1.4.5) of the compoent. While they both work (most of the time), there are some differences.
1) The 1.4.3 version will time-out (randomly) on the ftp.ListNames line.
2) The 1.4.5 version will break (randomly) on the (a) using (StreamReader sr = new StreamReader(ms)) and (b) while (sr.Peek() > 0) lines. When I say break, I mean that the debugger just drops out of the method without throwing an exception.
Here is the code that is being used in the sample application.
-Kent
FtpDLX ftp = null;
try {
ftp = new FtpDLX();
ftp.Authentication = Authentications.Password;
ftp.Hostname = txtServer.Text;
ftp.Login = txtAccountName.Text;
ftp.Password = txtAccountPassword.Text;
ftp.TransferMode = TransferModes.Binary;
ftp.LicenseKey = ???????? ;
ftp.Blocking = true;
ftp.StrictHost = true;
ftp.Passive = false;
ftp.Protocol = Protocols.FTP;
ftp.Timeout = 150;
ftp.Connect();
List<string> lst = new List<string>();
using (MemoryStream ms = new MemoryStream()) {
ftp.ListNames(txtLocation.Text, ms);
if (ftp.LastError != null) throw ftp.LastError;
ms.Seek(0, SeekOrigin.Begin);
using (StreamReader sr = new StreamReader(ms)) {
while (sr.Peek() > 0) {
string tmpFileName = sr.ReadLine();
MessageBox.Show( File: + tmpFileName);
if (!tmpFileName.StartsWith( . ) && !tmpFileName.EndsWith( / ) && Utils.FilePatterns.IsMatch(tmpFileName, txtPattern.Text)) lst.Add(tmpFileName);
}
}
}
for (int i = 0; i < lst.Count; i++) {
string localFile = Path.Combine(txtDownloadTo.Text, lst);
string remoteFile = string.Format( {0}/{1} , txtLocation.Text, lst);
MessageBox.Show( Downloading: + remoteFile);
ftp.GetFile(txtDownloadTo.Text, remoteFile);
MessageBox.Show( Fetched: + lst);
}
} catch (Exception ex) {
MessageBox.Show(ex.Message);
} finally {
if (ftp != null) {
try { if (ftp.State != States.Disconnected) ftp.Disconnect(); } catch { }
try { ftp.Dispose(); } catch { }
}
}
Re: Timeout Property
by wodDamir, Monday, July 06, 2009, 22:38 (5620 days ago) @ kdcarlisle
Kent,
A NullReference means that an object was accessed that had Null as it's value.
Can you please try stepping thru your code line-by-line and check where exactly you get NullReference?
Regards,
Damba
Re: Timeout Property
by kdcarlisle, Tuesday, July 07, 2009, 16:42 (5619 days ago) @ wodDamir
Damba,
Unfortunately I can not step through the production code. The code that I created is a powershell cmdlet that snaps into the MS Powershell framework. These cmdlets are then access via the powershell scripts. While I can debug the script, I have not found a way to debug the cmdlet itself. I originally tested all the cmdlet methods against a testing application. I could go into the code and add a bunch of debug statements to write out to a log file and then let it go.
The part that I am confused about is that the exception is being thrown and crashing the system. The code that calls the various FTP methods is wrapped in a try/catch block and any exceptions that are thrown should be caught here (which would then be written to the job's session log file).
I will see what information I can gather and post back soon.
-Kent
Re: Timeout Property
by woddrazen, Tuesday, July 07, 2009, 18:39 (5619 days ago) @ kdcarlisle
Kent,
Looking at your code I see that you read ListNames output in memory stream and then convert it to string.
Why don't you use ListItem Property instead? ListItem Property hold string result from ListNames and ListDir Method.
Can you maybe try something like this without memory stream:[code]...
ftp.ListNames(txtLocation.Text);
string tmpFileName = ftp.ListItem;
MessageBox.Show( File: + tmpFileName);
...[/code]
Let us know how it goes.
Drazen
Re: Timeout Property
by kdcarlisle, Wednesday, July 08, 2009, 17:03 (5618 days ago) @ woddrazen
Drazen,
I am currnetly testing the .ListItem property and working on breaking out the multiple file names. In my initial test I noticed that sometimes the .ListItem property is blank (which it should not be). I am testing against the same server (using same credentials) and same location. There are a set of test fiels out there specially for this test. Most of the time I get the list, but sometimes I do not. I changed up the code that was using the MemoryStream and I get the same results (every once in a while I get nothing back). The process is not timing out, it just doesn't return anything. Is there something that I do to further troubleshoot this issue?
-Kent
Re: Timeout Property
by woddrazen, Wednesday, July 08, 2009, 17:22 (5618 days ago) @ kdcarlisle
Kent,
Can you maybe duplicate this on our side? Is there any chance you can send us exact type and version of server you are using?
In one of your replies you send us some code snippet. Can we use that code for test on our side?
How many files you have in that folder and what's approximate size of files? Also you are running this code in loop or not?
Sorry for all my questions but we have to isolate your problem somehow and try to resolve it.
Drazen
Re: Timeout Property
by kdcarlisle, Thursday, July 09, 2009, 01:44 (5618 days ago) @ woddrazen
Drazen,
The FTP server is: Titan FTP Server / Version 5.39 Build 389 / Enterprise Edition
There are 4 files and 1 folder within my location
prefeed [folder]
CSFRAG2.txt.pgp (1,287 KB)
CSFRAG2.txt.pgp-38102535 (1,333 KB)
CSFRAG2.txt.pgp-38107984 (1,334 KB)
testfile.txt (1,334 KB)
The code is not running in a loop.
You can use the code snippet that I posted.
If you have powershell I can send you the cmdlets that use the component.
If you are wanting to access (via ftp) into our server, then I will have to get permission again (plus I will need your ip for our firewall).
I know that we did not pay for the source, but would it be possible to get a pdb file so that I could step into the .ListNames method and see what is happening. I would then delete the pdb file when done and would not decompile it (nor post it anywhere). I completly understand if you do not want to do this, but I thought I'd throw it out there for your consideration.
-Kent
Re: Timeout Property
by woddrazen, Thursday, July 09, 2009, 23:24 (5617 days ago) @ kdcarlisle
Kent,
Problem with empty result of ListNames Property should be fixed.
Can you please request update and try it again?
Let us know if you have more issues to duplicate.
Drazen
Re: Timeout Property
by kdcarlisle, Sunday, July 12, 2009, 18:25 (5614 days ago) @ woddrazen
Drazen,
Thanks for the update. I applied it (when you posted) and have been monitoring the system. All of the issues have gone away.
Thanks again for all the help on this!!
Kent