How to download a file from the internet? [SOLVED]

Helper

On 24/09/2014 at 07:40, xxxxxxxx wrote:

No idea.
I needed a way to download a text file from the web using C++ a couple of years ago. So I went searching for the code to do that. And this is what I found.
I tried it and it worked. Then I never used it again.

I do recall that this code was hard to find.
Most of the really robust URL stuff for C++ seems to be sold in commercial packages.

-ScottA

Helper

On 24/09/2014 at 11:08, xxxxxxxx wrote:

Oh, man!!!
Python makes this so, so easy.
Well, I will search more and more.
Thank you, Scott.

Helper

On 24/09/2014 at 22:25, xxxxxxxx wrote:

You might want to take a look at libcurl.

http://curl.haxx.se/libcurl/

And if you like the C++ way of doing things then also curlpp, which makes use of libcurl.

https://code.google.com/p/curlpp/

I wrote a blog post about this years ago when I was writing a system to download files from cloud services. This may help you on your way to compiling both libcurl and curlpp.

http://www.gamelogicdesign.com/2012/05/21/a-c-library-for-communicating-with-amazon-s3/

Once you have libcURL integrated into your plugin then you issue HTTP requests (PUT, POST, GET etc..) to communicate with the webservice (website) you are connecting with. So for your situation you would send a HTTP POST with the login and password, get a message back to tell you if that was successful or not and then you can make another call to GET a file.

Kent

Helper

On 25/09/2014 at 01:47, xxxxxxxx wrote:

Thank you for the answer, Kent.
Is this cross platform?
I mean, I usually develop in Xcode and then send my source code to Visual Studio to compile for Win32 and Win64.

Helper

On 25/09/2014 at 04:09, xxxxxxxx wrote:

I already found out that I have "curl" included in Xcode.
So, I should use that.
However, all the samples (not that many, actually) that I find online use the curl library to download a URL directly to a file.
I don't want that. I want to do it like I do in python. I simply need to read the content of a small, simple .txt file that is online. Just 10 characters.
So, what I want is to access a .txt file that is inside a username/password protected folder.
I have the URL address, the username and the password. I just need to read the content of the file in the URL address.
Why is it that all the C++ code I find is soooooooo complicated?
My python code is:

  
import urllib2   
from urllib2 import URLError, HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler, install_opener, build_opener   
  
def check_online(url,username,password) :   
        
     try:   
          password_mgr.add_password(None, url, username, password)   
          opener = build_opener(HTTPBasicAuthHandler(password_mgr))   
          file = opener.open(url+"/myfile.txt")   
          content=file.read()   
          file.close()   
  
     except:   
          content=""   
  
     return content

So, so simple. Why must it be so complicated in C++?

Helper

On 25/09/2014 at 09:27, xxxxxxxx wrote:

Well, I reached as far as:

  
#include <stdio.h>   
#include <curl/curl.h>   
  
CURL *curl;   
CURLcode res;   
char* usrpass="my_username:my_password";   
char* pointer="1234567890";   
        
curl = curl_easy_init();   
if(curl) {   
     curl_easy_setopt(curl, CURLOPT_URL, "http://my_url_pointing_to_the_file");   
     curl_easy_setopt(curl, CURLOPT_USERPWD , usrpass);   
  
     curl_easy_setopt(curl, CURLOPT_READDATA, pointer);   
     curl_easy_setopt(curl, CURLOPT_READFUNCTION, NULL);   
             
     /* Perform the request, res will get the return code */   
     res = curl_easy_perform(curl);   
     /* Check for errors */   
     if(res != CURLE_OK)   
          fprintf(stderr, "curl_easy_perform() failed: %s\n",   
                         curl_easy_strerror(res));   
             
     /* always cleanup */   
     curl_easy_cleanup(curl);   
}

But, when building, I get:

Undefined symbols for architecture x86_64:
"_curl_easy_init", referenced from:
      PluginStart() in main.o
"_curl_easy_setopt", referenced from:
      PluginStart() in main.o
"_curl_easy_perform", referenced from:
      PluginStart() in main.o
"_curl_easy_strerror", referenced from:
      PluginStart() in main.o
"_curl_easy_cleanup", referenced from:
      PluginStart() in main.o
ld: symbol(s) not found for architecture x86_64

Also, I get "Conversion from string literal to 'char *' is deprecated"

Can someone help me out with this?

Helper

On 25/09/2014 at 18:39, xxxxxxxx wrote:

The reason that C++ is 'so complicated' is that it is literally the OOP extension of C which is a very low-level procedural language. C is basically a step above Assembler (directly programming the CPU) and C++ is not much further away. Remember that Python is interpreted while C++ is compiled into executable code (directly executable on the CPU). The choice of C/C++ is speed since executable code is almost always faster than interpreted code.

On the issue with curl, have you included a lib or dylib to the project?

Helper

On 25/09/2014 at 22:10, xxxxxxxx wrote:

I installed curl and your code works fine for me.
All I had to do was change the URL to some text file on Google that I could grab as a test.
I'm guessing that you probably unzipped the curl files to your HD. But you never actually built them?
curl needs to be "built" so that you get a .dll from it.
The instructions for building are in: winbuild/BUILD.WINDOWS.txt
I built mine using this: nmake /f Makefile.vc mode=dll

When you build curl it creates a new folder called "builds"
In there you'll find a bin folder with libcurl.dll in it.
That .dll file has to be copied to your VS project's Debug and Release folders or else they won't run.
QT users know about this sort of thing because they also need to copy some QT .dlls to their plugin's output folders if not using static builds.
It's a fairly common practice. So try to get comfortable with it.

Also. You have to set up your project's properties in VS:
-In the VC++ Directories section you have to set the paths to your curl folders in the "Include Directories" and "Library Directories".
-In the Linker->Input section. You have to open the "Additional Dependencies" option.
And paste this into the top window: libcurl.lib
BTW: If you're wondering where I got libcurl.lib from. I got it from the builds/lib folder which you should see after you build it.

This is similar to how I make all of my C4D plugins too.
I do it all from scratch. I don't copy and paste and then delete what's not needed as Maxon recommends.
I do it all completely from scratch every single time. Because when you download libraries from the web. You'll need to know how all the setting work in VS.
There's no such thing as copy and paste out there in the "real world".

One thing that you're going to notice is that people who write C++ code rarely communicate clearly to the average user.
And people who write these libraries are the worst at communication.
They have coded stuff for so long that they have completely forgotten what it's like to be a new or casual C++ user.

This is why C++ is thought of as "too hard".
It's not hard at all. It's only hard when people can't communicate clearly and properly.
There's files all over the place, and several things needing to be set up in VS for it all to work.
This stuff has to be communicated to the users clearly. And C++ programmers are the worst of the worst when it comes to communicating things clearly.

-ScottA

Helper

On 25/09/2014 at 23:55, xxxxxxxx wrote:

I have noticed that information is very unclear.
Cryptic sentences and, above all, almost all assume that is a veteran programmer reading them. Newbies try to read them and can't make anything out of it.
I have dozens of browser tabs with stuff related to curl but most of them I can't decipher properly

Well, I guess I have something to start with, now. Oh, by the way, my code was mostly what I could come up with after reading a LOT about curl.

However, I start my development in Xcode. I assume that I need to compile the curl library for Mac and compile for Windows too, when I take my code to VS.
So, I still need to find out how to compile it for Mac, first.
Actually, on my Mac, I was just including the curl library that is already in my system. But I guess it is not enough.
Well, off I go, trying to find out how to compile it for Mac.

Helper

On 03/10/2014 at 03:21, xxxxxxxx wrote:

Hello again, Scott.

I have it all working fine now in MacOS X. It was not easy but it is working perfectly now.
Now I was trying to build the Win32 and Win64 versions.
So, while trying to build the curl libraries, I got this:

c:\Users\ruibatista\Desktop\curl-7.38.0\winbuild>nmake Makefile.vc mode=dll VC=10

configuration name: libcurl-vc10-x86-release-dll-ipv6-sspi-winssl
cl.exe /O2 /DNDEBUG /MD /I. /I ../lib /I../include /nologo /W3 /EHsc /DW
IN32 /FD /c /DBUILDING_LIBCURL /I"../../deps/include" /DUSE_WIN32_IDN /DWANT_ID
N_PROTOTYPES /DUSE_IPV6 /DUSE_WINDOWS_SSPI /DUSE_SCHANNEL /Fo"..\builds\libcur
l-vc10-x86-release-dll-ipv6-sspi-winssl-obj-lib/file.obj" ..\lib\file.c
file.c
../include\curl/curlbuild.h(152) : fatal error C1083: Cannot open include file:
'sys/socket.h': No such file or directory
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0
\VC\BIN\cl.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0
\VC\BIN\nmake.exe"' : return code '0x2'
Stop.

Must I have the curl folder in some specific place?
Compiling curl in MacOS was so easy. I only had to type one command and everything worked fine the first time

Helper

On 03/10/2014 at 03:23, xxxxxxxx wrote:

Also, one of the options is:

MACHINE=<x86 or x64>

Does this mean that, if I want to build Win32 and Win64 version I have to perform two compilations? Will they not conflict?

(but this only matters if I manage to compile it, of course, and I can't even do that now)

Helper

On 03/10/2014 at 08:01, xxxxxxxx wrote:

The 32 bit version of anything will typically compile fine when your compiler is making a 64 bit program. So it should be safe to set that option to x64. Assuming that you're building a 64 bit plugin.
If you're configuration is set to build a Debug | 64 or Release | 64 plugin. Then that option needs to be set to x64.
If you're configuration is set to build a Debug | Win32 or Release | Win32 plugin. Then that option needs to be set to Win32.

The one I installed was: curl 7.38.0. It wasn't the MSVC version.
It's ok to rename it to "curl" to make the path shorter and easier to use.

Here's the instructions I wrote for myself in case I forget how I did it

Installation instructions  
///////////////////////////////////////////////////////////////////////////////////////////////////////////  
  
-Unzip to c:\curl  
-Build curl to generate the .dll and .lib files needed  
 I used this in the VC console to build it: nmake /f Makefile.vc mode=dll  
  
  
-Set up VS to use them:  
  Put libcurl.lib in the Linker->Input "Additional Dependencies" top window  
  Put C:\curl\builds\libcurl-vc-x86-release-dll-ipv6-sspi-winssl\include\curl   in "Include Directories"   
  Put C:\curl\builds\libcurl-vc-x86-release-dll-ipv6-sspi-winssl\lib            in "Library Directories"  
  
  
  
//////////////////////////////////////////////////////////////////////////////////////////////////////////////  
  
Using the examples  
  
  
-If you can't run your code because of a "libcurl.dll not found" error  
 You might need to copy the builds/bin/libcurl.dll file to your Degug & Release folders to fix that  
  
  
  
NOTE: In VS you will get lots of 'cannot convert from void..' errors from the examples  
    This is because they need to be casted to work  
    Examples: chunk.memory = (char* )malloc(1);  
              mem->memory = (char* )realloc(mem->memory, mem->size + realsize + 1);

-ScottA

Helper

On 03/10/2014 at 11:13, xxxxxxxx wrote:

Once again, thank you. I managed to do it with a installer from here (http://www.confusedbycode.com/curl/)
But, if it weren't for you indications, I would never guess that those changes in the Prject Properties were required.
However, now I'm getting these errors:

1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\xlocale(323) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\xlocale(323) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2143: syntax error : missing ')' before 'constant'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2143: syntax error : missing ';' before 'constant'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2059: syntax error : ')'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2238: unexpected token(s) preceding ';'
1>
1>Build FAILED.
1>
1>Time Elapsed 00:00:02.10
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

I didn't included and <msxml.h>
How come it is returning those errors?

Helper

On 03/10/2014 at 11:54, xxxxxxxx wrote:

I'm afraid I have no idea about those.

One thing that I've noticed is that VS will throw missing ';' or missing ')'; error messages if you forget the closing brace '}' somewhere in your code.
VS apparently doesn't know how to check your scoping {} braces very well. And it throws those kinds of misleading error messages. And sends you on a wild goose chase if you forgot to close your scope with an '}' in your code.

I doubt that's why you're getting those error though.

-ScottA

Helper

On 03/10/2014 at 12:00, xxxxxxxx wrote:

I also doubt that because I do all my initial development in Xcode and Xcode is excellent at balancing { }, " " and ( )
Damn!!! It would be so fine if it was just a matter of moving the source and res folder from my Mac project to the VS project

Helper

On 05/10/2014 at 15:16, xxxxxxxx wrote:

1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\xlocale(323) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\xlocale(323) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2143: syntax error : missing ')' before 'constant'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2143: syntax error : missing ';' before 'constant'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2059: syntax error : ')'
1>c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(1436) : error C2238: unexpected token(s) preceding ';'

I think in one of your files you are including <windows.h> or some other header that includes this (probably from the curl libs). Then after than include you are then including some c4d headers, perhaps "c4d.h". You will now get conflicts with windows when compiling.

Basically the issue is that you should not mix any non C4D header includes in a file that uses C4D.

What I would do is have a seperate CPP file that does all the calls to libcurl. Do not have any C4D includes in this at all. Create a header file for that CPP that has standard C++ variables. DownloadFile(const char *url, int len). Then implement this method in the CPP file to call all the libcurl required functions, this CPP file should not call any C4D code or have any C4D headers in it. Its just for libcurl. There is nothing else at all in the header file either.

Then in your C4D files you can include this header to call the methods. Or you could ignore the header file and forward declare these methods directly in your c4d file.

Helper

On 05/10/2014 at 16:02, xxxxxxxx wrote:

Oh shoot. You're right Kent.
Sorry about that Rui. I forgot to tell you about the order of the includes.
The C4D includes always need to be placed after your STL stuff.

If you have your C4D includes listed first. That could explain the errors Rui.
Try putting #include <curl.h> ~~after~~ before your C4D includes.
It works fine me.

-ScottA

Helper

On 05/10/2014 at 16:13, xxxxxxxx wrote:

I did found that it had to be something to do with the #include <curl.h> just a while ago. I was commenting out each line of #include (and correspondent dependent code) and the errors stopped when I commented the #include <curl.h> line.
I do have my #include <curl.h> after my #include "c4d.h".
My #include's are like this:

  
#include "c4d.h"   
  
#include "serialization.h"   
  
#include <stdio.h>   
#include <curl.h>   
#include <sys/stat.h>   
#include <time.h>   
  
#include <sstream>   
#include <fstream>   
#include <iostream>   
#include <string>   
#include <stdlib.h>

I will try out creating an individual .cpp file for all my curl stuff tomorrow. It is quite late here and I must go to bed now.
I will let you guys know how it went. Thank you all

Helper

On 05/10/2014 at 16:38, xxxxxxxx wrote:

Note: Order of header inclusion can be very important. Typically, you want to include standard C++ headers first, SDK headers next, and then third-part headers last (just as a rule of thumb). Even then, you should juxtapose them during testing/debugging to resolve issues.

Another note (that I learned the hard way) : Never, ever include headers in headers. Include them in the source file (.cpp). It avoids any deeply-nested issues that may occur.

Helper

On 05/10/2014 at 16:54, xxxxxxxx wrote:

Doh!
I meant put your STL stuff before your c4d includes.
Sorry about that. My brain must be fried from trying to use the Mudbox SDK.
If you think the C4D SDK is bad...try using the Mudbox SDK. :zipper_mouth:

curl compiles fine for me when it's inside of a C4D plugin.
However. I do get unresolved errors from this code if I use it inside of a plugin: curl = curl_easy_init();
Apparently VS can't find this method for some strange reason.

When I tested your code and the curl examples. I tested them in a raw C++ project. Not inside of a C4D plugin.
There seems to some more hoops to solve if you want to run the curl code inside you're plugin.
You might be better off running it in a raw C++ project. Then having your C4D plugin execute your Raw C++ curl code.

Fun...ain't it.

-ScottA