Commit Graph

18 Commits

Author SHA1 Message Date
Shyotl
1c627317ec No longer include llerrorlegacy.h. Updated llstl to include deletion utilites. 2015-06-19 03:54:20 -05:00
Aleric Inglewood
8c6d51cb71 No longer support DEBUG_CURLIO when libcwd isn't installed. 2014-08-25 17:58:45 +02:00
Aleric Inglewood
ec436b0e3b Improve upload_finished detection for HTTP requests.
We detect the end of an upload when libcurl informs us
that it is no longer interested in knowing if a socket
is writable (which it tells us cause we're doing the call
to select(), by means of a call to MultiHandle::socket_callback).

However, this does not *always* work.
The debug output might look like this:

[...initialization and first part of a TLS handshake...]
0x7f759f0ad700   CURL        :   select(55, {17, 39, 46, 47, 48, 49, 50, 51, 52, 54}, NULL, NULL, timeout = 1 ms) = 7
0x7f759f0ad700   CURLTR      :   curl_multi_socket_action((CURLM*)0x7f758c002920, 46, CURL_CSELECT_IN, <unfinished>	<--- Calling curl_multi_socket_action (socket 46 is readable)
0x7f759f0ad700   CURLIO      0x13fce7b0       * SSLv3, TLS change cipher, Client hello (1):
0x7f759f0ad700   CURLIO      0x13fce7b0       S> 1 bytes
0x7f759f0ad700   CURLIO      0x13fce7b0       * SSLv3, TLS handshake, Finished (20):
0x7f759f0ad700   CURLIO      0x13fce7b0       S> 16 bytes
0x7f759f0ad700   CURLIO      0x13fce7b0       * SSL connection using AES256-SHA
0x7f759f0ad700   CURLIO      0x13fce7b0       * Server certificate:
0x7f759f0ad700   CURLIO      0x13fce7b0       *          subject: C=US; ST=California; L=San Francisco; O=Linden Lab, Inc.; CN=*.agni.lindenlab.com; emailAddress=root@lindenlab.com
0x7f759f0ad700   CURLIO      0x13fce7b0       *          start date: 2012-09-20 22:55:47 GMT
0x7f759f0ad700   CURLIO      0x13fce7b0       *          expire date: 2015-09-20 22:55:47 GMT
0x7f759f0ad700   CURLIO      0x13fce7b0       *          issuer: C=US; ST=California; L=San Francisco; O=Linden Lab, Inc.; OU=Linden Lab Certificate Authority; CN=Linden Lab Certificate Authority; emailAddress=ca@lindenlab.com
0x7f759f0ad700   CURLIO      0x13fce7b0       *          SSL certificate verify ok.
0x7f759f0ad700   CURLIO      0x13fce7b0       * STATE: PROTOCONNECT => DO handle 0x7f758d62d810; (connection #2)
0x7f759f0ad700   CURLIO      0x13fce7b0       H< POST /cap/d67d4540-4504-b3d1-1d26-112010e88bd8 HTTP/1.1\r\nHost: sim10480.agni.lindenlab.com:12043\r\nAccept: */*\r\nAccept-Encoding: deflate, gzip\r\nContent-Type: application/llsd+xml\r\nX-SecondLife-UDP-Listen-Port: 32837\r\nConnection: keep-alive\r\nKeep-alive: 300\r\nContent-Length: 333\r\n\r\n
0x7f759f0ad700   CURLIO      0x13fce7b0       * STATE: DO => DO_DONE handle 0x7f758d62d810; (connection #2)
0x7f759f0ad700   CURL        :       Entering BufferedCurlEasyRequest::curlProgressCallback(0x13fce7b0, 0, 0, 333, 0)	<=== THIS IS NEW: total to upload: 333 bytes, total uploaded: 0 bytes.
0x7f759f0ad700   CURLIO      0x13fce7b0       * STATE: DO_DONE => WAITPERFORM handle 0x7f758d62d810; (connection #2)
0x7f759f0ad700   CURL        :       Entering BufferedCurlEasyRequest::curlProgressCallback(0x13fce7b0, 0, 0, 333, 0)
0x7f759f0ad700   CURLIO      0x13fce7b0       * STATE: WAITPERFORM => PERFORM handle 0x7f758d62d810; (connection #2)
0x7f759f0ad700   CURL        :       Entering BufferedCurlEasyRequest::curlProgressCallback(0x13fce7b0, 0, 0, 333, 0)
0x7f759f0ad700   CURL        :       Entering BufferedCurlEasyRequest::curlProgressCallback(0x13fce7b0, 0, 0, 333, 0)
0x7f759f0ad700   CURL        :       Entering BufferedCurlEasyRequest::curlProgressCallback(0x13fce7b0, 0, 0, 333, 0)
0x7f759f0ad700   CURLTR      :       curl_easy_getinfo((CURL*)0x13fcf320, CURLINFO_PRIVATE, 0x7f759f0abdb0) = CURLE_OK
0x7f759f0ad700   CURL        :       Entering MultiHandle::socket_callback((CURL*)0x13fcf320, 46, CURL_POLL_INOUT, 0x7f758c001508, 0x7f758c2ce4e0) [CURLINFO_PRIVATE = 0x13fce7b0]
0x7f759f0ad700   CURL        :         CurlSocketInfo::set_action(CURL_POLL_IN --> CURL_POLL_INOUT) [0x13fce7b0]	<=== WATING FOR WRITING BUT ALREADY WAITING FOR INPUT!
0x7f759f0ad700   CURL        0x13fce7b0          reset_lowspeed: mLowSpeedClock = 138530973390; mStalled = -1
0x7f759f0ad700   CURLTR      :   <continued> {11}) = 0									<--- Returning from curl_multi_socket_action
0x7f759f0ad700   CURL        :   select(58, {17, 39, 46, 47, 48, 49, 50, 51, 52, 54}, {36, 46, 57}, NULL, timeout = 756 ms) = 5
0x7f759f0ad700   CURLTR      :   curl_multi_socket_action((CURLM*)0x7f758c002920, 46, CURL_CSELECT_OUT, <unfinished>	<=== SOCKET AVAILABLE FOR WRITING
0x7f759f0ad700   CURLIO      0x13fce7b0       * additional stuff not fine transfer.c:1037: 0 0
0x7f759f0ad700   CURLIO      0x13fce7b0       D< 333 bytes: "<llsd><map><key>folders</key><array><map><key>fetch_folders</key>...etc"	<=== WRITING *ALL* DATA
0x7f759f0ad700   CURL        :       Entering BufferedCurlEasyRequest::curlProgressCallback(0x13fce7b0, 0, 0, 333, 333)	<=== THIS IS NEW, DETECT END OF UPLOAD
0x7f759f0ad700   CURL        0x13fce7b0          upload_finished: mStalled set to Time_10ms (138530973390) + 6000 (60 seconds)
0x7f759f0ad700   CURL        :       Entering BufferedCurlEasyRequest::curlProgressCallback(0x13fce7b0, 0, 0, 333, 333)
0x7f759f0ad700   CURL        :   select(58, {17, 39, 46, 47, 50, 52, 54}, {36, 46, 57}, NULL, timeout = 0 ms) = 8
0x7f759f0ad700   CURLTR      :   curl_multi_socket_action((CURLM*)0x7f758c002920, 46, CURL_CSELECT_IN|CURL_CSELECT_OUT, <unfinished>
0x7f759f0ad700   CURLIO      0x13fce7b0       * HTTP 1.1 or later with persistent connection, pipelining supported
0x7f759f0ad700   CURLIO      0x13fce7b0       H> HTTP/1.1 200 OK\r\n

...and the server already replies without a call to MultiHandle::socket_callback between
writing and reading, because the socket was still (already) waiting for input anyway
(since it was waiting for the SSL/TLS handshake before in this case, but
it also happens when the data has to be written in more than one writes).

Using the progress callback however, we are able to detect the end of the upload right
after writing, anyway.
2013-11-24 19:13:17 +01:00
Aleric Inglewood
34b921a101 Disable llassert(!mUploadFinished)
This fires so little that it isn't worth anymore to look into it
(the only problem resulting from this is a wrong timeout on one
of the http requests); and therefore no longer makes sense to
have it triggered in alpha builds at all, since those do not
produce enough debug output to understand what caused it in the
first place.
2013-06-12 21:21:41 +02:00
Aleric Inglewood
45e6b7975f First version of HTTP bandwidth throttling.
Adds throttling based on on average bandwidth usage per HTTP service.
Since only HTTP textures are using this, they are still starved by other
services like inventory and mesh dowloads. Also, it will be needed to
move the maximum number of connections per service the to the PerService
class, and dynamically tune them: reducing the number of connections is
the first thing to do when using too much bandwidth.

I also added a graph for HTTP texture bandwidth to the stats floater.
For some reason the average bandwidth (over 1 second) look almost like
scattered noise... weird for something that is averaged...
2013-04-24 22:04:21 +02:00
Aleric Inglewood
734d2e658d Rename HTTPTimeout::sClockCount (in clock ticks) to sTime_10ms (in 10ms units).
Note that HTTPTimeout::sClockWidth is no longer used for HTTPTimeout (as
if it's value became a constant of 0.01, the fraction 10ms / 1s).
A new variable, HTTPTimeout::sClockWidth_10ms is used to calculate
sTime_10ms (only).

Also HTTPTimeout::mStalled and HTTPTimeout::mLowSpeedClock changed units
to the same as of sTime_10ms (time since epoch in 10ms units).
2013-04-09 22:27:23 +02:00
Aleric Inglewood
bb948ce6d5 Use host:port as key for the "PerHost" request queue, instead of just the hostname.
Rationale: LL is doing all throttling per service (host:port), not per
service hostname. Also, textures and capabilities use the same host: the
sim you are connected to. Splitting the queues up on a per-service basis
will stop the textures from blocking a capability request.
2013-04-09 04:32:36 +02:00
Aleric Inglewood
b20886a481 Allow TOS redirect. Fix upload finished detection when redirecting.
This fixes https://code.google.com/p/singularity-viewer/issues/detail?id=705

Adds 'bool redirect_status_ok(void) const { return true; }' to LLIamHere,
because it's ok to receive a 302 status there. Likewise added to LLIamHereVoice,
because that has the same comment in its error() method.

Also fixes the problem that if two redirects occur on a row, then the
upload_finished detection asserted because it would detect the third
time that libcurl turned off writing to the socket as a failure (the
second time wasn't a problem because mUploadFinished was reset upon
receiving the first 302 header, but not upon receiving the second
header).
2013-03-25 04:41:07 +01:00
Aleric Inglewood
5a8308109b Lowered CurlTimeoutLowSpeedLimit from 56 kB/s to 7 kB/s.
This value really IS in bytes/s (not for the total), and apparently 56
kB/s is too optimistic. The value was used by LL for transfers that went
beyond the total download time (2 minutes?), adding enough seconds per
received bytes to the timeout to allow a download at 56 kB/s to finish.

Our meaning is different however: we time out immediately and whenever
the download drops below this speed.

Perhaps a better algorithm is where the speed demand is based on the
total size of the download, but I'm not sure we always know the size of
downloads at this point.
2013-03-10 17:45:39 +01:00
Aleric Inglewood
7dfef31469 Possible upload timeout improvement.
When uploading finishes, but is not detected, the timeout should be for
"reply delay", the time that the server takes before it replies, and not
CurlTimeoutLowSpeedTime. This patch adds code that takes this failure
into account (which happened only ONCE for me on Metropolis while flying
around and using trickle (not sure if that is relevant), so it's not
that likely to improvement anything in practise. Note that it is
detected by an assertion when it happens, so that we can safely assume
it normally never happened on SL).

* Generalized PUT / POST configuration by adding
  CurlEasyRequest::setPut, which now also supports keep-alive (which
  still isn't used).
* Upload content length is now stored in CurlEasyRequest::mContentLength
* CurlEasyRequest::has_stalled() now return false if it was possbile
  that the 'upload finished' detect failed AND calls upload_finished()
  itself in that case, so it is no longer 'const'.
* If low speed is detect exactly when the last bytes are being attempted
  to be sent (unlikely scenario), then the upload gets 4 more seconds
  after which is switches to CurlTimeoutReplyDelay.
* Added EDoesAuthentication and EAllowCompressedReply to replace
  booleans, for readability and type-safety, as did EKeepAlive. Note
  that this change inverts the meaning of the compression related parameter.
* Unrelated: removed an unnecessary #include "llurlrequest.h" from
  llxmlrpcresponder.h
2013-03-10 16:43:15 +01:00
Aleric Inglewood
96d302aa42 Be more verbose about internal (curl / timeout) errors.
Translates the CURLE_WRITE_ERROR into what it really means: that the low
speed check failed.
2013-03-09 05:26:52 +01:00
Aleric Inglewood
b6cbed838f Assertion bug fix for libcwd with dc::curl on and curlio off. 2013-02-23 21:37:07 +01:00
Aleric Inglewood
78bb94eb42 Bug fix: remove remainder of test code 2013-02-23 19:36:18 +01:00
Aleric Inglewood
dc3f33717c Fixed assertion in HTTPTimeout::lowspeed
mTotalBytes had the meaning of "the sum of all elements in the vector",
but was reset to 0 at the start of a transaction. The meaning is now
changed to "total number of bytes in the vector that belong to the
current transaction".
2013-02-14 00:09:42 +01:00
Aleric Inglewood
c6c343a45b Compile bug fixes when -DDEBUG_CURLIO 2013-02-12 23:44:55 +01:00
Aleric Inglewood
3a414b21fa Temporary workaround for release. 2013-01-13 02:18:00 +01:00
Aleric Inglewood
e99a906f54 Testsuite part 1: make it compile alone. 2013-01-13 01:58:49 +01:00
Aleric Inglewood
fc4b02cee8 Moved HTTPTimeout to it's own files. 2013-01-12 23:00:21 +01:00