Huge files over 500 MB? (General questions)

Hi there,

I own WodWebServer and need to use it to upload really, really large files, like over 500 MB and eventually over 2 GB!
I've been testing it will 100 MB files uploaded to localhost.
It works with 100 MB files (albeit pretty slowly) but with the 200 MB one, it seems to hang on:
f = e.user.Request.Uploads.Count

Are there any tips or limitations for dealing with files this large? Or does it just not support files of this size?

Thanks!!

Also please remove my email from the last post. It is my work email and I don't want to get spam since this post is public!

Thanks!

Hi there,

I own WodWebServer and need to use it to upload really, really large files, like over 500 MB and eventually over 2 GB!
I've been testing it will 100 MB files uploaded to localhost.
It works with 100 MB files (albeit pretty slowly) but with the 200 MB one, it seems to hang on:
f = e.user.Request.Uploads.Count

Are there any tips or limitations for dealing with files this large? Or does it just not support files of this size?

Thanks!!

Update: It seems to be whenever the Uploads stucture is accessed with a large file (250 MB or more), it hangs for 15 minutes or so and then resumes. I'm assuming it is parsing the file data into the Uploads structure on-demand. Once it is done that for the first time, future access is fine.

Are there plans to add support for large files (250 MB-2GB+)? And if not, are there any other ways to work with large files, (aside from parsing the body file directly?). I can't use HTTP Put, I need to to HTTP Post.

Code Examples Where the Freeze happens
h = u.Request.Uploads(0)
For Each m As WODWEBSERVERLib.IWebRequestUploadPart In u.Request.Uploads

Hi Simon,

Hangs as in stops receiving new requests, or it hangs completely?

Did you perhaps try setting Threads property?

Regards,
Damba

The problem is not even the hanging but the fact that it takes 15 to 20 minutes or more to process the large file.

I am not at my programming PC, but I recall it hangs completely and the processor spikes.

Hi Simon,

Hangs as in stops receiving new requests, or it hangs completely?

Did you perhaps try setting Threads property?

Regards,
Damba

Hi. I will check it out, but please note that wodWebServer needs to parse the file line by line and search for boundaries that split files that are uploaded. So, if file is large, this can be pretty lenghty process.

Unlike applications, wodWebServer component doesn't have the ability to postpone processing or fire it in separate thread, since programmer can access the property any time he wishes.

Kreso

Hi,

Thanks. I do think that it would be most efficient to check for the ending part of a file as it is streamed from the client initially.

Another technique is the just seek to the end and assume that there is one large file being uploaded, get the last 5000 bytes and then you know where it starts and stops.

Can you let me know your plan to addressing the issue of large files? Because my own coding plan is dependent on that.

Hi. I will check it out, but please note that wodWebServer needs to parse the file line by line and search for boundaries that split files that are uploaded. So, if file is large, this can be pretty lenghty process.

Unlike applications, wodWebServer component doesn't have the ability to postpone processing or fire it in separate thread, since programmer can access the property any time he wishes.

Kreso

Hi. Yes, this will be addressed, I'm just giving out my thoughts about it.

I cannot handle the way you suggested. We cannot assume only one file is uploaded so we can't just seek to the end of uploaded data. We must search *all* uploaded data.

It will take few days since even small tests take several minutes.

Kreso

That's great, I really appreciate it.

Good point about the multiple files problem.

Please let me know when the update is available via this thread (I'm subscribed).

Hi. Yes, this will be addressed, I'm just giving out my thoughts about it.

I cannot handle the way you suggested. We cannot assume only one file is uploaded so we can't just seek to the end of uploaded data. We must search *all* uploaded data.

It will take few days since even small tests take several minutes.

Kreso

One idea to mention:

Ideally, I'd never need to recopy or resave the 1GB file after the upload, since it is so large. Instead, I plan on moving the file to the folder I want on the drive. (Moving files is very fast when it is to the same drive.)

One possible solution is, in the requestheaders, to specify a folder and save each attachment as a separate file. The file name can be the specified name or a UUID or something. That would make it easy for programmers to work with tho I don't know if that's the best solution for you.

That's great, I really appreciate it.

Good point about the multiple files problem.

Please let me know when the update is available via this thread (I'm subscribed).

Hi. Yes, this will be addressed, I'm just giving out my thoughts about it.

I cannot handle the way you suggested. We cannot assume only one file is uploaded so we can't just seek to the end of uploaded data. We must search *all* uploaded data.

It will take few days since even small tests take several minutes.

Kreso

But it's not that easy. WHen file(s) arrive, they are saved all together in temporary file. Then they are extracted from this file to separate files, and this can take ages.

Have you considered some other way for uploads? What is your programming environment? Who is your client, regular browser or some component?

Kreso

Its actually for a file server so it needs to be sent from a standard web browser. Needs to be done through a web server and not FTP unfortunately.
So the idea is to have them upload the file and then move it to their home folder. This is why I want to avoid copying the large file.

They will also be able to download that same file after logging in.

Often they'll want to share the file they uploaded just after finishing the upload.

I asked about programming environment to see if you have threading capabilities, so perhaps parsing can be done in separate thread.

But I'm also considering to provide AsyncSave method, that would take the file and export it to real file in separate thread, so main process doesn't freeze. It will still take 15-20 seconds, but your app would be responsive in the meantime.

Kreso

O .. I am using VB.NET 2008, so I do have threading. I use the ActiveX.

Personally for me, AsyncSave wouldn't solve the problem. Best would be something to let me stream the file from wodWebServer as it loads, similar to a BinaryReader. (Ex: I could then send it as a response to another client as well as a file at the same time. Even tho the whole file might not be saved, since they are downloading slowly anyways, it would be fine.)

Also, note sure if it is possible, but could there be support for 2GB+ files like with WodFTPServer? [I have that too.]

I asked about programming environment to see if you have threading capabilities, so perhaps parsing can be done in separate thread.

But I'm also considering to provide AsyncSave method, that would take the file and export it to real file in separate thread, so main process doesn't freeze. It will still take 15-20 seconds, but your app would be responsive in the meantime.

Kreso

By the way, when I say as it loads I mean *after* the entire Request file is uploaded and saved to disk, I could have something like:

Dim dataBuffer As Byte()
Dim FileReader As New BinaryReader(e.user.Request.Uploads(0).getFileAsStream)

dataBuffer = BinaryReader.ReadBytes(1000000)
My.Computer.FileSystem.WriteAllBytes( CustomerFile.Zip , dataBuffer, True)

If you did this with the .NET version of WodWebServer, then I would be willing to pay a reasonable price to upgrade to the .NET version, since I don't know if BinaryReaders exist outside of that.

O .. I am using VB.NET 2008, so I do have threading. I use the ActiveX.

Personally for me, AsyncSave wouldn't solve the problem. Best would be something to let me stream the file from wodWebServer as it loads, similar to a BinaryReader. (Ex: I could then send it as a response to another client as well as a file at the same time. Even tho the whole file might not be saved, since they are downloading slowly anyways, it would be fine.)

Also, note sure if it is possible, but could there be support for 2GB+ files like with WodFTPServer? [I have that too.]

I asked about programming environment to see if you have threading capabilities, so perhaps parsing can be done in separate thread.

But I'm also considering to provide AsyncSave method, that would take the file and export it to real file in separate thread, so main process doesn't freeze. It will still take 15-20 seconds, but your app would be responsive in the meantime.

Kreso

Tho actually I'm not sure if that's even possible given the structure. I guess you'd first need to analyze the Request to see where the file(s) even start!

Only way to do this without locking up would be with an unusual structure, like:
u.Request.ManualUploadProcessing = True

do

dim result as boolean = u.Request.Uploads.processNextUpload

if not result then exit do

c = u.Request.Uploads.current
c.getName
c.getContentType
dim br as new BinaryReader(c.getDataStream)
....

loop

By the way, when I say as it loads I mean *after* the entire Request file is uploaded and saved to disk, I could have something like:

Dim dataBuffer As Byte()
Dim FileReader As New BinaryReader(e.user.Request.Uploads(0).getFileAsStream)

dataBuffer = BinaryReader.ReadBytes(1000000)
My.Computer.FileSystem.WriteAllBytes( CustomerFile.Zip , dataBuffer, True)

If you did this with the .NET version of WodWebServer, then I would be willing to pay a reasonable price to upgrade to the .NET version, since I don't know if BinaryReaders exist outside of that.

O .. I am using VB.NET 2008, so I do have threading. I use the ActiveX.

Personally for me, AsyncSave wouldn't solve the problem. Best would be something to let me stream the file from wodWebServer as it loads, similar to a BinaryReader. (Ex: I could then send it as a response to another client as well as a file at the same time. Even tho the whole file might not be saved, since they are downloading slowly anyways, it would be fine.)

Also, note sure if it is possible, but could there be support for 2GB+ files like with WodFTPServer? [I have that too.]

I asked about programming environment to see if you have threading capabilities, so perhaps parsing can be done in separate thread.

But I'm also considering to provide AsyncSave method, that would take the file and export it to real file in separate thread, so main process doesn't freeze. It will still take 15-20 seconds, but your app would be responsive in the meantime.

Kreso

Simon,

just to keep you posted about the progress. We're adding AutoParse property that will allow you to force wodWebServer save uploaded files immediately, during upload. It will probably introduce new event where you can set destination filename too. When used, Request.Body will be short of actualy body data, since it will be immediately exported to destination files.

I believe this will solve your problem. File size will be irrelevant. I hope to have some solution perhaps even today.

Kreso

That does solve it thanks!
Could I choose whether or not to parse?
Like, if the are accessing a script that is *not* for uploading, if possible could I still access e.user.Request.Posted and e.user.Request.QueryVariables

Thanks!!! I appreciate the hard work!

Simon,

just to keep you posted about the progress. We're adding AutoParse property that will allow you to force wodWebServer save uploaded files immediately, during upload. It will probably introduce new event where you can set destination filename too. When used, Request.Body will be short of actualy body data, since it will be immediately exported to destination files.

I believe this will solve your problem. File size will be irrelevant. I hope to have some solution perhaps even today.

Kreso

Simon,

if you're anxious to try, you can request update and try it out. We are still testing it, but it's online.

You will find new propety AutoParse in Uploads collection. Inside ReqhestHeadersDone you should set it to True.

It will fire one or more UploadFilename where you should set Part.Filename property to destination file.

Let me know how it goes!
Kreso

You make decision if you will parse in RequestHeadersDone event, where you have all the headers and can see what's arriving.

Kreso

That's perfect!!!!

Thanks so much!

You make decision if you will parse in RequestHeadersDone event, where you have all the headers and can see what's arriving.

Kreso

Going to update and try out the current version now, I'll let you know how it goes...

Hi there,

I didn't get any callback to the UploadFilename sub, even after I uploaded a file. Here's the code I used. Am I missing something?
I did use the COM control and implemented the interface. I am getting callbacks to other functions such as RequestDone just fine. I'm in VB.NET 2008...

Imports System.IO
Imports System.Text

Public Class Form1

Implements WODWEBSERVERCOMLib.IwodWebNotify

Dim WebServer1 As WODWEBSERVERCOMLib.wodWebServerCom

Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click

WebServer1 = New WODWEBSERVERCOMLib.wodWebServerCom
WebServer1.LicenseKey = HIDDEN 'Taken out for forum post
WebServer1.Notification = Me
WebServer1.DocumentRoot = C: esthtml
WebServer1.MyHostname = localhost
WebServer1.AllowBrowsing = True
WebServer1.Threads = WODWEBSERVERCOMLib.ThreadTypes.FullThreads
WebServer1.Start()
End Sub

Public Sub CGIStart(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser, ByVal FullPath As String, ByVal Environment As WODWEBSERVERCOMLib.WebHeaders) Implements WODWEBSERVERCOMLib.IwodWebNotify.CGIStart

End Sub

Public Sub CGIStop(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser, ByVal success As Boolean) Implements WODWEBSERVERCOMLib.IwodWebNotify.CGIStop

End Sub

Public Sub RequestDone(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser) Implements WODWEBSERVERCOMLib.IwodWebNotify.RequestDone

End Sub

Public Sub RequestHeaders(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser) Implements WODWEBSERVERCOMLib.IwodWebNotify.RequestHeaders

User.Request.Uploads.AutoParse = True

End Sub

Public Sub ResponseDone(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser) Implements WODWEBSERVERCOMLib.IwodWebNotify.ResponseDone

Dim ss As String = User.Request.FullURI

End Sub

Public Sub RunScript(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser, ByVal ScriptData As String, ByRef ResponseData As String) Implements WODWEBSERVERCOMLib.IwodWebNotify.RunScript

End Sub

Public Sub StateChange(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser, ByVal OldState As WODWEBSERVERCOMLib.StatesEnum) Implements WODWEBSERVERCOMLib.IwodWebNotify.StateChange

End Sub

Public Sub UploadFilename(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser, ByVal Part As WODWEBSERVERCOMLib.WebRequestUploadPart) Implements WODWEBSERVERCOMLib.IwodWebNotify.UploadFilename
'Code does not even execute to here
Part.Filename = c:igone.zip
End Sub

Public Sub UserAuthenticate(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser, ByVal AuthType As WODWEBSERVERCOMLib.WebAuthenticationTypes, ByRef Action As WODWEBSERVERCOMLib.WebActions) Implements WODWEBSERVERCOMLib.IwodWebNotify.UserAuthenticate

End Sub

Public Sub UserConnected(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser, ByRef Action As WODWEBSERVERCOMLib.WebActions) Implements WODWEBSERVERCOMLib.IwodWebNotify.UserConnected

End Sub

Public Sub UserDisconnected(ByVal Owner As WODWEBSERVERCOMLib.wodWebServerCom, ByVal User As WODWEBSERVERCOMLib.WebUser) Implements WODWEBSERVERCOMLib.IwodWebNotify.UserDisconnected

End Sub
End Class

Going to update and try out the current version now, I'll let you know how it goes...

You seem to be doing all ok. I will try to duplicate your code.

BTW if you check Uploads.Part(0).Filename, does it point to some temporary file? YOu can check that in RequestDone event.

If it does, then it's working ok, only problem is UploadFilename event.

Kreso

Thanks for the prompt response!

I actually can't access Uploads.Part at all. It seems I still have the old WebRequestUploads interface. Maybe that's the issue?

Ex: if I just do:
dim d as WebRequestUploads
then there's no .Part

Just in case it was .NET messing up the ActiveX... I also tried it in VB6 with the COM control and got the same thing. Might be easier to try out that code, so just in case here it is:

Option Explicit
Dim WithEvents Http1 As wodWebServerCom
Implements IwodWebNotify

Private Sub Form_Load()
Set Http1 = New wodWebServerCom

Set Http1.Notification = Me

Http1.DocumentRoot = App.Path

On Error Resume Next
Http1.Start
If Err Then
MsgBox There was an error: & Err.Description, vbOKOnly, Error
End If
End Sub

Private Sub IwodWebNotify_CGIStart(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser, ByVal FullPath As String, ByVal Environment As WODWEBSERVERCOMLib.IWebHeaders)

End Sub

Private Sub IwodWebNotify_CGIStop(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser, ByVal success As Boolean)

End Sub

Private Sub IwodWebNotify_RequestDone(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser)

User.Response.Delivery = Automatic
Exit Sub

End Sub

Private Sub IwodWebNotify_RequestHeaders(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser)

User.Request.Uploads.AutoParse = True
End Sub

Private Sub IwodWebNotify_ResponseDone(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser)

End Sub

Private Sub IwodWebNotify_RunScript(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser, ByVal ScriptData As String, ResponseData As String)

End Sub

Private Sub IwodWebNotify_StateChange(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser, ByVal OldState As WODWEBSERVERCOMLib.StatesEnum)

End Sub

Private Sub IwodWebNotify_UploadFilename(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser, ByVal Part As WODWEBSERVERCOMLib.IWebRequestUploadPart)
Part.FileName = c:myfile.pdf
End Sub

Private Sub IwodWebNotify_UserAuthenticate(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser, ByVal AuthType As WODWEBSERVERCOMLib.WebAuthenticationTypes, Action As WODWEBSERVERCOMLib.WebActions)

End Sub

Private Sub IwodWebNotify_UserConnected(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser, Action As WODWEBSERVERCOMLib.WebActions)

End Sub

Private Sub IwodWebNotify_UserDisconnected(ByVal Owner As WODWEBSERVERCOMLib.IwodWebServerCom, ByVal User As WODWEBSERVERCOMLib.IWebUser)

End Sub

Just in case it was .NET messing up the ActiveX
- Sorry I meant.. with .NET I had previously used the ActiveX implementation before switching to COM, so to ensure there's no interference with any .NET stuff I did it in VB6 above.

Hi,

I just tried your sample, and it works for me. UploadFileName event is normally triggered.

As for part, what Kreso meant was to check User.Uploads.Item(0).Filename or User.Uploads(0).Filename .

There is no Part member in WebUploads object.

Also, can you please check what your Version property returns?

regards,
Damba

Version is: 1.5.3.295
Checking the User.Uploads(0).Filename and getting back to you.

Hi,

I just tried your sample, and it works for me. UploadFileName event is normally triggered.

As for part, what Kreso meant was to check User.Uploads.Item(0).Filename or User.Uploads(0).Filename .

There is no Part member in WebUploads object.

Also, can you please check what your Version property returns?

regards,
Damba

It actually says that User.Request.Uploads.Count is 0 after uploading a large file.

When I access the
fn = User.Request.Uploads(0).FileName
I get invalid procedure call or argument

Since it works for you, maybe it is something left out of the distribution file?

Here's the HTML upload code I use:
<html>

<head>
<meta http-equiv= Content-Type content= text/html; charset=windows-1252 >
<meta name= GENERATOR content= Microsoft FrontPage 4.0 >
<meta name= ProgId content= FrontPage.Editor.Document >
<title>UPLOAD</title>
</head>

<body>

<form enctype= multipart/form-data action= uploader.php method= POST >
<input type= hidden name= MAX_FILE_SIZE value= 100000 />
UPLOAD
<p>
Choose a file to upload: <input name= uploadedfile type= file /><br />
<input type= submit value= Upload File />
</form>

</body>

</html>

Simon,

Can you please contact us at techsupport@weonlydo.com and I'll send you the attachment with the sample that is working on our side.

Regards,
Damba

Email sent, looking forward to it.

I also tried uninstalling, restarting, then re-installing, then restarting and it still seems to not save with AutoParse.