Archived Forum Post
Index of archived forum posts
Question:
Archived Forum PostQuestion:
Is this true or did i miss something?
I mean what is this API worth anyhow if there is no shared background thread? I can start a thread with a few lines of code myself so where is the benefit from it. I have to open thousand connections to satisfy the 100MBit connection of my web spider. This is eating up a lot of resources and for small GET with failed IF-modified requests i'm sure there is a lot of overhead.
I know i should have checked this during 30days trial but honestly i'm very very disappointed if i'm right. Is there any way to implement real background thread/asynchronous processing myself?
It raises the overall question in my mind: Do you see performance as a serious design goal or is it just easy useability?
The purpose of the async background method calls is to satisfy the needs of those that might be developing applications in certain programming languages where it is difficult to create a background thread, and the need is simply to do an HTTP request in the background without blocking the UI. It's not necessarily a feature for a high-performance web spider. For every developer trying to create a high-performance web spider, there are a thousand others with a far simpler needs.
It sounds like what you really need is to decouple the request-sending from the response-receiving. This is the only way a single thread can have multiple requests ongoing at once. The Chilkat HTTP API methods both send and receive the response, and the entire request-response may be more complicated than a single request followed by a single response. There could be authentication (NTLM for example which requires a 3-step process), or 100-continue, or redirects, etc. These things are all handled automatically.
Given your needs, you probably want to use the Socket API. GET requests are simple to form. You can create your own thread and then implement some scheme where you send requests and get responses. HTTP responses, unless chunked and/or gzipped, are MIME. Your request could indicate that you don't want a gzipped response. I can't recall, but it may be possible to indicate you don't want chunked responses. In that case, given that HTTP requests and responses are simply a start line followed by MIME, you could use Chilkat MIME to parse each response.