I need to parse the debug output of libcurl for this, and choose
to not do that for a Release build - at which point it makes little
sense to even do it for a Debug build, since also the Alpha release
are 'Release'. Even though it would probably have no impact on FPS,
there would be like two persons looking at that number and understanding
it.
This make more sense when comparing the threshold against 2 times the
maximum allowed total connections.
As a result, textures won't stall when there are around 128 mesh
requests queued.
Prepares for throttling number of connections per capability type.
The value is current left at AIPerService::mConcurrectConnections,
so not having an effect yet.
This splits the AIPerService queue up into four queues: one for each
"capability type". On Second Life that doesn't make a difference in
itself for textures because the texture service only serves one
capability type: textures. Other services however can serve two types,
while on Avination - that currently only has one services for everything
- this really makes a difference because that single service now has
four queues.
More importantly however is that the administration of how many requests
are in the "pipeline" (from approving that a new HTTP request may be
added for given service, till curl finished it) is now per capability
type (or service/capabitity type pair actually). This means downloads of
a certain capability type (textures, inventory, mesh, other) will no
longer stall because unapproved requests cluttered the queue for a given
service.
Moreover, before when a request did finished, it would only look for a
new request in the queue of the service that just finished. This simple
algorithm worked when there were no 'PerSerice' objects, and only one
'Curl' queue: because if anything was queued that that was because there
were running requests, and when one of those running requests finished
it made sense to see if one of those queued requests could be added now.
However, after adding multiple queues, one for each service, it could
happen that service A had queued requests while only requests from
service B were actually running: only requests of B would ever finish
and the requests of A would be queued forever.
With this patch the algorithm is to look alternating first in the
texture request queue and then in the inventory request queue - or vice
versa, and if there are none of those, look for a request of a different
type. If also that cannot be found, look for a request in another
service. This is still not optimal and subject to change.
If AIPerService::add_queued_to fails because a new request is throttled,
then do not add the request to the end of the queue, nor remove it from
the queue: do nothing: it makes no sense to move the request to the back
because they all belong to the same service and all of them will be
either throttled or not.
Note: Still need to fix that in this case we should look in queues of
other services.
The inventory bulk fetch is not thread-safe, so the it doesn't start
right away, causing the approvement not to be honored upon return from
post_approved (formerly post_nb).
This patch renames wantsMoreHTTPReqestsFor to approveHTTPRequestFor,
and has it return NULL or a AIPerService::Approvement object.
The latter is now passed to the CurlEasyHandle object instead of just a
boolean mQueueIfTooMuchBandwidthUsage, and then the Approvement is
honored by the state machine right after the request is actually added
to the command queue.
This should avoid a flood of inventory requests in the case
approveHTTPRequestFor is called multiple times before the main thread
adds the requests to the command queue. I don't think that actually ever
happens, but I added debug code (to find some problem) that is so damn
strictly checking everything that I need to be this precise in order to
do that testing.
* Removed the 'RequestQueue' from other PerServiceRequestQueue occurances
in the code.
* Made wantsMoreHTTPRequestsFor and checkBandwidthUsage threadsafe (by
grouping the static variables of AIPerService into thread ThreadSafe
groups.
Most notably getMesh (the only one possibly using any significant
bandwidth), but in general every type of requests that just have to
happen anyway and in the order they are requested: they are just passed
to the curl thread, but now the curl thread will queue them and hold
back if the (general) service they use is loaded too heavily.
Don't pass arguments to wantsMoreHTTPRequestsFor, but use globals in
llmessage: AIPerService::sHTTPThrottleBandwidth125 and
AIPerService::sNoHTTPBandwidthThrottling instead.
This is needed later on.
Adds throttling based on on average bandwidth usage per HTTP service.
Since only HTTP textures are using this, they are still starved by other
services like inventory and mesh dowloads. Also, it will be needed to
move the maximum number of connections per service the to the PerService
class, and dynamically tune them: reducing the number of connections is
the first thing to do when using too much bandwidth.
I also added a graph for HTTP texture bandwidth to the stats floater.
For some reason the average bandwidth (over 1 second) look almost like
scattered noise... weird for something that is averaged...
Note that HTTPTimeout::sClockWidth is no longer used for HTTPTimeout (as
if it's value became a constant of 0.01, the fraction 10ms / 1s).
A new variable, HTTPTimeout::sClockWidth_10ms is used to calculate
sTime_10ms (only).
Also HTTPTimeout::mStalled and HTTPTimeout::mLowSpeedClock changed units
to the same as of sTime_10ms (time since epoch in 10ms units).
After commit things compile again :).
The HTTP bandwidth throttling is not yet implemented. I'll put a
temporary fix back the next commit that just does it the "old way"...
This includes also non-texture requests (not sure if it's worth it to
explicitely separate this data for just textures).
The texture console now prints: HTTP:c/q/a/r
Where, c = number of 'add' request commands in the command queue
(minus the number of 'remove' request in the command queue and not
taking into account the command_being_processed, nor entirely being
thread-safe with regard to adding requests to 'q' or 'a' next: it is
possible that a request is no longer counted in 'c' but not yet is added
to 'q' or 'a'.
q = number of queued (throttled) requests (by the curl thread).
a = number of actually added requests (to the multi handle).
r = last returned value of 'running handles' by libcurl.
Obviously, c and q should be small (0 or 1) most of the time and a and r
should be equal and maxed out. This turns out to be the case.
Renamed PerHostRequestQueue and PerHostRequestQueuePtr to
AIPerHostRequestQueue and AIPerHostRequestQueuePtr respectively and
moved them to global namespace. This in preparation for using them in
the texture fetcher (as function of the hostname of the url that is
contructed there).
This new variable is updated to contain the total number of requests in
MultiHandle::mAddedEasyRequests. It can be a static because MultiHandle
is a singleton (there is only one curl thread) and it has to be an
(atomic) static because we don't want to need to take the lock on
MultiHandle for accessing this variable.
This adds a size (command_queue_st::size) to the ThreadSafe deque<Command> that was
AICurlPrivate::command_queue that is kept equal to the number of 'add'
commands in the deque minus the number of 'remove' commands in the
deque. The deque itself is now command_queue_st::commands.
Replaces LLTextureFetch::getNumHTTPRequests.
Returns AICurlInterface::Stats::running_handles.
This is work in progress that temporarily doesn't compile because
LLTextureFetch::getNumHTTPRequests is still being used somewhere.
This fixes https://code.google.com/p/singularity-viewer/issues/detail?id=705
Adds 'bool redirect_status_ok(void) const { return true; }' to LLIamHere,
because it's ok to receive a 302 status there. Likewise added to LLIamHereVoice,
because that has the same comment in its error() method.
Also fixes the problem that if two redirects occur on a row, then the
upload_finished detection asserted because it would detect the third
time that libcurl turned off writing to the socket as a failure (the
second time wasn't a problem because mUploadFinished was reset upon
receiving the first 302 header, but not upon receiving the second
header).
This is now necessary since the curl thread no longer syncs with the
main thread: it is possible that a request finishes after a texture
fetch thread was shot down but before curl was stopped, and curl
calling BufferedCurlEasyRequest::processOutput while objects that the
responder uses were already destructed (most notably
LLTextureFetch itself).
This value really IS in bytes/s (not for the total), and apparently 56
kB/s is too optimistic. The value was used by LL for transfers that went
beyond the total download time (2 minutes?), adding enough seconds per
received bytes to the timeout to allow a download at 56 kB/s to finish.
Our meaning is different however: we time out immediately and whenever
the download drops below this speed.
Perhaps a better algorithm is where the speed demand is based on the
total size of the download, but I'm not sure we always know the size of
downloads at this point.
When uploading finishes, but is not detected, the timeout should be for
"reply delay", the time that the server takes before it replies, and not
CurlTimeoutLowSpeedTime. This patch adds code that takes this failure
into account (which happened only ONCE for me on Metropolis while flying
around and using trickle (not sure if that is relevant), so it's not
that likely to improvement anything in practise. Note that it is
detected by an assertion when it happens, so that we can safely assume
it normally never happened on SL).
* Generalized PUT / POST configuration by adding
CurlEasyRequest::setPut, which now also supports keep-alive (which
still isn't used).
* Upload content length is now stored in CurlEasyRequest::mContentLength
* CurlEasyRequest::has_stalled() now return false if it was possbile
that the 'upload finished' detect failed AND calls upload_finished()
itself in that case, so it is no longer 'const'.
* If low speed is detect exactly when the last bytes are being attempted
to be sent (unlikely scenario), then the upload gets 4 more seconds
after which is switches to CurlTimeoutReplyDelay.
* Added EDoesAuthentication and EAllowCompressedReply to replace
booleans, for readability and type-safety, as did EKeepAlive. Note
that this change inverts the meaning of the compression related parameter.
* Unrelated: removed an unnecessary #include "llurlrequest.h" from
llxmlrpcresponder.h
