Adds a std::map for hostname (or urls) --> PerHostRequestQueue
objects. The latter keeps track of the number of added curl easy
requests and decides if a new request should be throttled or
not, as well as provides the queue to queue throttled requests.
At the moment CurlConcurrentConnectionsPerHost is set to 16,
because things really don't work without LL supporting connection
reuse if we limit it to 2. CurlConcurrentConnectionsPerHost is
also set to non-persistent so that we can easily change it in the future
(once we decide on it's final value it can be set to persistent).
Moved AICurlPrivate::Stats to AICurlInterface::Stats and added several
counters to keep track of the number of existing instances of
respectively AICurlEasyRequest, AICurlEasyRequestStateMachine,
BufferedCurlEasyRequest, ResponderBase and
ThreadSafeBufferedCurlEasyRequest.
As idle statemachines aren't in any list, it's not possible
(without adding that list) to delete them. I don't think
that there are any active statemachines left at the end
of flush anyway, but killing them doesn't much sense if
we can't get them all: there will always be statemachines
left: those that were idle at the moment the viewer was
quit.
Since we changed CurlResponderBuffer to be derived from CurlEasyRequest
and therefore changed it's name to BufferedCurlEasyRequest, we should
also rename AICurlResponderBufferEvents to
AIBufferedCurlEasyRequestEvents.
This commit also fixes C++ comment in several places to reflex the
previous name change.
Rename check_run_count to check_msg_queue, because the whole 'run count'
approach is flawed anyway (the author of libcurl told me that THE way
to check for finished curl handles is to just call curl_multi_info_read
every time: it's extremely fast. Any test that attempts to avoid that
call is nonsense anyway.
The reason the assertion failed might have been caused by the fact
that we're comparing the current number of easy handles with the
number of running handles of 'a while ago'. It is possible that a
easy handle was removed in the meantime.
In order to check if that hypothesis is right, I moved the assertion
to directly below the call to curl_multi_socket_action where it
should hold. If this new assertion doesn't trigger than the hypothesis
was right and this is fixed.
Every curl transaction is a AICurlEasyRequestStateMachine which has a
AICurlEasyRequest as member, which is a reference counting pointer to
a ThreadSafeBufferedCurlEasyRequest. And now BufferedCurlEasyRequest is
derived from CurlEasyRequest which is derived from CurlEasyHandle, but
neither are used separatedly.
* Moved Responder stuff to LLHTTPClient.
* Renamed LLHTTPClient::Responder to LLHTTPClient::ResponderWithResult.
* Deleted LLHTTPClientAdapter and LLHTTPClientInterface.
* Renamed AICurlInterface::TransferInfo to AITransferInfo and moved it
to llhttpclient.h
* Removed 'CURLcode code' argument from completed_headers.
* Removed LLCurlRequest and replaced it's last usage with LLHTTPClient API calls.
* Deleted dead code.
* Renamed all the get4/post4/put4/getByteRange4 etc, back to their
original name without the '4'.
* Moved LegacyPolledResponder::mCode to ResponderBase::mCode.
* Added a parameter to ResponderBase::finished (and pubError*) to set mCode.
* Renamed ResponderBase::decode_body to decode_llsd_body and added ResponderBase::decode_raw_body.
* Use LegacyPolledResponder::finished instead of LegacyPolledResponder::completed_headers
to set remaining cached values.
* Fixed assertion in case of -DCWDEBUG and upload finish detection failure in case of HEAD
method (mDebugIsGetMethod -> mDebugIsHeadOrGetMethod).
* Add XmlTreeInjector : support for LLXmlTree.
* Split BlockingResponder into BlockingLLSDResponder and BlockingRawResponder.
* Final blocking responders are now: BlockingLLSDPostResponder, BlockingLLSDGetResponder
and BlockingRawGetResponder.
* Added LLHTTPClient::blockingGetRaw
* Got rid of hipporestrequest.* -- and fixed hippogridmanager.cpp to use
LLHTTPClient::blockingGetRaw instead, and fixed llviewermessage.cpp to use
AICurlInterface::ResponderWithCompleted and decode_raw_body instead of
HippoRestHandlerRaw and LLHTTPClient::get4 instead of HippoRestRequest::get5.
Renamed AICurlInterface::Responder to AICurlInterface::ResponderBase,
but without the virtual 'event' methods.
Derived from that: Responder and ReponderWithCompleted, where the
first defines result = 0, ErrorWithContent and error, and the latter
completedRaw and completed.
Added HttpClient::IgnoreBody, derived from Responder and implementing
'result' doing nothing; HttpClient::Ignore is now derived from
IgnoreBody and defines the still pure virtual getHTTPTimeoutPolicy.
Added ResponderBase::decode_body, which is now the sole place
where the code makes the decision wether some response data might be
LLSD or not based on the http status result. Before it just tried
to decode everything as LLSD, which seems a bit nonsense.
ResponderWithCompleted::completed no longer does anything, since
classes derived from ResponderWithCompleted are expected to override it,
or never call it by overriding completedRaw.
Entry point is now ResponderBase::finished = 0, instead of
completedRaw, where ResponderWithCompleted implements finished by
called completedRaw, but Responder doesn't: that directly calls
result/errorWithContent/error. Or, for the hack ResponderAdapter,
the entry points are pubResult/pubErrorWithContent.
Those are now the ONLY public methods, so more confusion.
mFinished is now set in all cases.
As a result of all that, it is no longer possible to accidently
pass a responder to ResponderAdapter that would break because it
expects completed() and completedRaw() to be called.
Added LLBufferArray::writeChannelTo.
Fixed bug for BlockingResponder::body (returned reference to temporary).
LLSDMessage::ResponderAdapter now allows a "timeoutpolicy" name
to be passed (not doing so results in the default timings), so
that the timeout policy of the used responder is retained.
Fixed llfasttimerview.cpp to test LLSDSerialize::fromXML() to return
a positive value instead of non-zero, because it may return -1 when the
parsing fails (three places).
Removed LLHTTPClient::Responder as base class from
LLFloaterRegionDebugConsole completely: it isn't a responder!
Several other responder classes were simplified a bit in order to
compile again with the above changes.
Slightly more robust, adds one boolean to all responders, 99%
of which don't need that though, and an extra call redirection,
but well... We might need it this way when I add the possibility
to abort a transfer.
In days of usage this has never happened before, but apparently
it's possible. The solution chosen is to create the AIHTTPTimeout
object on the fly when it doesn't exist and let it be picked up
later when the CurlSocketInfo for the transfer is created.
This fixes the problem that existed with received headers:
The server sends some headers ("set-cookie") more than once
in the same reply, which cannot be stored in std::map.
The old code just ignored the additional cookies, while
curlthreading3 (since the introduction of AIHTTPHeaders)
caused an assertion.
AIHTTPReceivedHeaders is written around a std::multimap
and allows to retrieve multiple headers with the same key.
Also, it is case insensitive so that if a server sends
"Content-Type" it will still find it (the viewer looks for
"content-type").
Minor changes like comment fixes and addition of accessors
that will be needed for future commits.
Also removed Responder::fatalError as it was never used.
Moved CURLOPT_ENCODING from CurlEasyRequest::setPost_raw, and
CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST from
CurlResponderBuffer::prepRequest, to LLURLRequest::configure,
enabling the debug setting NoVerifySSLCert for the latter
two to work as follows: old behavior if "NoVerifySSLCert"
is not set, and check neither if it is set. However, if
the (new) bool mIsAuth is set the behavior of LLXMLRPCTransaction::Impl::init
is used. This is so in a next commit we can replace
LLXMLRPCTransaction with LLURLRequest: LLXMLRPCTransaction::Impl::init
will be removed. For the same reason, when the new boolean
mNoCompression is set then CURLOPT_ENCODING is set to "identity",
otherwise the old behavior (of clearing it) is used.
* Moved DoutCurlEasy and DoutCurlEasyEntering from aicurl.cpp
to aicurl.h and renamed them to DoutCurl and DoutCurlEntering
respectively.
* Moved the callback functions from aicurl.cpp to aicurlthread.cpp.
* In CurlEasyRequest, renamed timeout_timings to print_curl_timings
and mTimeoutLowercaseHostname to mLowercaseHostname.
* Put all remaining CurlEasyRequest::mTimeout* variables and
timeout_* methods in curlthread::HTTPTimeout, stripping them
of said prefix, and moved the definition to aicurlprivate.h.
Added a ThreadSafeCurlEasyRequest* member and a get_lockobj()
method so to that class so we can still use DoutCurl /
DoutCurlEntering. timeout_add_easy_request was removed completely
and reimplemented as the constructor of HTTPTimeout.
timeout_has_stalled was renamed to HTTPTimeout::has_stalled,
but also reimplemented as CurlEasyRequest::has_stalled.
* CurlEasyRequest::mRequestFinalized was removed and it's
functionality taken over by CurlEasyRequest::mTimeoutPolicy.
* Fixed the indentation of struct Stats, class CurlEasyHandle
and class CurlEasyRequest.
* Added CurlEasyRequest::set_timeout_opts
* Added CurlSocketInfo::mTimeout (LLPointer<HTTPTimeout>).
* mUploadFinished is now reset in HTTPTimeout::data_received,
this was needed because "HEAD /something" header-only
messages triggered upload_finished (unlike "GET ..."),
and in combination with redirection that caused an assert.
Use the better understandable alias CURLOPT_HTTPHEADER in debug output
Bug fix - finally!
This fixes a bug I've been looking for a week.
By accidently calling get_clock_count() for the mTimeoutLowSpeedClock
'event', the time difference 'now' - 'event' becomes negative, which
should be impossible; the result being that timeout_low_speed
stalled for 24 seconds while looping over the full 32 bit. That in
turn made SSL handshakes of libcurl fail, which seemed to be
impossible since the calls to libcurl had not changed!
