o ee@sLddlZddlZddlZdZdZdZejjZej dZ Gddde Z dS)Nconnect disconnectprobe reconnectc@seZdZdZGdddeZGdddeZGdddeZGdd d eZGd d d eZ Gd d d eZ GdddeZ ddZ ddZ ddZddZddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4Zd5d6Zd7d8Z d9d:Z!d;d<Z"d=d>Z#d?d@Z$dAdBZ%dCdDZ&dEdFZ'dGdHZ(dIdJZ)dKdLZ*dMdNZ+dOdPZ,dQdRZ-dSdTZ.dUS)V Reconnecta2A finite-state machine for connecting and reconnecting to a network resource with exponential backoff. It also provides optional support for detecting a connection on which the peer is no longer responding. The library does not implement anything networking related, only an FSM for networking code to use. Many Reconnect methods take a "now" argument. This makes testing easier since there is no hidden state. When not testing, just pass the return value of ovs.time.msec(). (Perhaps this design should be revisited later.)c@,eZdZdZdZeddZeddZdS)zReconnect.VoidVOIDFcCdSNfsmnowr r //usr/lib/python3/dist-packages/ovs/reconnect.pydeadline.zReconnect.Void.deadlinecCr r r r r r rrun2rzReconnect.Void.runN__name__ __module__ __qualname__name is_connected staticmethodrrr r r rVoid* rc@r)zReconnect.Listening LISTENINGFcCr r r r r r rr:rzReconnect.Listening.deadlinecCr r r r r r rr>rzReconnect.Listening.runNrr r r r Listening6rrc@r)zReconnect.BackoffBACKOFFFcCs |j|jSr ) state_enteredbackoffr r r rrFs zReconnect.Backoff.deadlinecCtSr )CONNECTr r r rrJrzReconnect.Backoff.runNrr r r rBackoffBrr#c@r)zReconnect.ConnectInProgress CONNECTINGFcCs|jtd|jS)N)rmaxr r r r rrRsz$Reconnect.ConnectInProgress.deadlinecCr!r  DISCONNECTr r r rrVrzReconnect.ConnectInProgress.runNrr r r rConnectInProgressNrr)c@r)zReconnect.ActiveACTIVETcCsJ|jr#t|j|j}||j}||ks|jdus|j|kr|S|dSdSN)probe_intervalr& last_activityrlast_receive_attempt)r rbase expirationr r rr^s   zReconnect.Active.deadlinecCs4td|j|t|j|jf||tjt S)Nz(%s: idle %d ms, sending inactivity probe) vlogdbgrr&r.r _transitionrIdlePROBEr r r rrsszReconnect.Active.runNrr r r rActiveZs r7c@r)zReconnect.IdleIDLETcCs>|jr|j|j}||ks|jdus|j|kr|S|dSdSr+)r-rr/)r rr1r r rrs   zReconnect.Idle.deadlinecCs"td|j||jdftS)NzE%s: no response to inactivity probe after %.3g seconds, disconnecting@@)r2errrrr(r r r rrszReconnect.Idle.runNrr r r rr5{s  r5c@r)zReconnect.Reconnect RECONNECTFcC|jSr )rr r r rrszReconnect.Reconnect.deadlinecCr!r r'r r r rrrzReconnect.Reconnect.runNrr r r rrrcCsd|_d|_d|_d|_d|_tj|_tj |_ ||_ d|_ ||_ d|_d|_||_d|_d|_||_d|_d|_d|_d|_dS)zCreates and returns a new reconnect FSM with default settings. The FSM is initially disabled. The caller will likely want to call self.enable() and self.set_name() on the returned object.voidr%@iFrN)r min_backoff max_backoffr-passiver2info info_levelrrstaterr r.last_connectedlast_disconnectedr/ max_triesbackoff_free_tries creation_timen_attempted_connectionsn_successful_connectionstotal_connected_durationseqnoselfrr r r__init__s( zReconnect.__init__cCs|rtj|_dStj|_dS)aIf 'quiet' is true, this object will log informational messages at debug level, by default keeping them out of log files. This is appropriate if the connection is one that is expected to be short-lived, so that the log messages are merely distracting. If 'quiet' is false, this object logs informational messages at info level. This is the default. This setting has no effect on the log level of debugging, warning, or error messages.N)r2r3rCrB)rOquietr r r set_quiets  zReconnect.set_quietcCr<r rrOr r rget_nameszReconnect.get_namecCs|dur d|_dS||_dS)zSets this object's name to 'name'. If 'name' is None, then "void" is used instead. The name is used in log messages.Nr=rS)rOrr r rset_names  zReconnect.set_namecCr<)zReturn the minimum number of milliseconds to back off between consecutive connection attempts. The default is 1000 ms.)r?rTr r rget_min_backoffzReconnect.get_min_backoffcCr<)zReturn the maximum number of milliseconds to back off between consecutive connection attempts. The default is 8000 ms.)r@rTr r rget_max_backoffrXzReconnect.get_max_backoffcCr<)aReturns the "probe interval" in milliseconds. If this is zero, it disables the connection keepalive feature. If it is nonzero, then if the interval passes while the FSM is connected and without self.activity() being called, self.run() returns ovs.reconnect.PROBE. If the interval passes again without self.activity() being called, self.run() returns ovs.reconnect.DISCONNECT.)r-rTr r rget_probe_intervalszReconnect.get_probe_intervalcC ||_dS)a,Limits the maximum number of times that this object will ask the client to try to reconnect to 'max_tries'. None (the default) means an unlimited number of tries. After the number of tries has expired, the FSM will disable itself instead of backing off and retrying.NrG)rOrGr r r set_max_triess zReconnect.set_max_triescCr<)zeReturns the current remaining number of connection attempts, None if the number is unlimited.r\rTr r r get_max_triesrXzReconnect.get_max_triescCsft|d|_|jrt|d|_nd|_|j|jkr|j|_|jtjkr/|j|jkr1|j|_dSdSdS)a,Configures the backoff parameters for this FSM. 'min_backoff' is the minimum number of milliseconds, and 'max_backoff' is the maximum, between connection attempts. 'min_backoff' must be at least 1000, and 'max_backoff' must be greater than or equal to 'min_backoff'.r%r>N)r&r?r@rDrr#r )rOr?r@r r r set_backoffs     zReconnect.set_backoffcCr[)zSets the number of connection attempts that will be made without backoff to 'backoff_free_tries'. Values 0 and 1 both represent a single attempt.N)rH)rOrHr r rset_backoff_free_tries s z Reconnect.set_backoff_free_triescCs|r td||_dSd|_dS)a%Sets the "probe interval" to 'probe_interval', in milliseconds. If this is zero, it disables the connection keepalive feature. If it is nonzero, then if the interval passes while this FSM is connected and without self.activity() being called, self.run() returns ovs.reconnect.PROBE. If the interval passes again without self.activity() being called, self.run() returns ovs.reconnect.DISCONNECT. If 'probe_interval' is nonzero, then it will be forced to a value of at least 1000 ms.r%rN)r&r-)rOr-r r rset_probe_intervals  zReconnect.set_probe_intervalcCr<)zaReturns true if 'fsm' is in passive mode, false if 'fsm' is in active mode (the default).)rArTr r r is_passive!rXzReconnect.is_passivecCsf|j|kr+||_|r|jtjtjfvs|s-|jtjkr/|r1||tjd|_dSdSdSdSdS)zConfigures this FSM for active or passive mode. In active mode (the default), the FSM is attempting to connect to a remote host. In passive mode, the FSM is listening for connections from a remote host.rN) rArDrr)r_Reconnect__may_retryr4r#r )rOrArr r r set_passive&s    zReconnect.set_passivecCs |jtjkS)zReturns true if this FSM has been enabled with self.enable(). Calling another function that indicates a change in connection state, such as self.disconnected() or self.force_reconnect(), will also enable a reconnect FSM.)rDrrrTr r r is_enabled5s zReconnect.is_enabledcCs4|jtjkr|r||tjd|_dSdSdS)zIf this FSM is disabled (the default for newly created FSMs), enables it, so that the next call to reconnect_run() for 'fsm' will return ovs.reconnect.CONNECT. If this FSM is not disabled, this function has no effect.rN)rDrrrcr4r#r rNr r renable<s zReconnect.enablecCs"|jtjkr||tjdSdS)zZDisables this FSM. Until 'fsm' is enabled again, self.run() will always return 0.N)rDrrr4rNr r rdisableFs zReconnect.disablecCs,|jtjtjtjfvr||tjdSdS)aIf this FSM is enabled and currently connected (or attempting to connect), forces self.run() to return ovs.reconnect.DISCONNECT the next time it is called, which should cause the client to drop the connection (or attempt), back off, and then reconnect.N)rDrr)r7r5r4rNr r rforce_reconnectLs zReconnect.force_reconnectcCsx|jtjtjfvr:|jtjtjfvr;|dkr%td|jt |fnq|t kr2| d|jnd| d|jn[|jtj kr\|dkrStd|jt |fnC| d|jn:|jtjkrk| d|jn+|j|jkr|jrwd}nd }|dkrtd |j|t |fn | d |j|f|jtjtjfvr||_|s||tjd S|jd kr|jd 8_d|_nr|jtjtjfvr|j|j|jks|jr|jrd|_nV|j|_nQ|j|jkr|j|_nF|j|jdkr|jd9_|jrd}nd}| d|j|jd|fn|j|jkr-|jr!d}nd}| d|j|f|j|_||tjd Sd S)a_Tell this FSM that the connection dropped or that a connection attempt failed. 'error' specifies the reason: a positive value represents an errno value, EOF indicates that the connection was closed by the peer (e.g. read() returned 0), and 0 indicates no specific error. The FSM will back off, then reconnect.rz%s: connection dropped (%s)z%s: connection closed by peerz%s: connection droppedz(%s: error listening for connections (%s)z#%s: error listening for connectionsz%s: connection closed by clientlisten connectionz%s: %s attempt failed (%s)z%s: %s attempt timed outNr,ztrying to listen againrz"%s: waiting %.3g seconds before %sr9z try to listenzF%s: continuing to %s in the background but suppressing further logging)rDrr#rr7r5r2warnrosstrerrorEOFrCrr r@rArFrcr4rHr.rEr?)rOrerrortype_actionr r r disconnectedVs        zReconnect.disconnectedcCsV|jtjkr)|jr|d|jn|j|jkr |d|j||tjdSdS)zTell this FSM that a connection or listening attempt is in progress. The FSM will start a timer, after which the connection or listening attempt will be aborted (by returning ovs.reconnect.DISCONNECT from self.run()).%s: listening...z%s: connecting...N) rDrr)rArCrr r@r4rNr r r connectings  zReconnect.connectingcCs2|jtjkr|d|j||tjdSdS)aTell this FSM that the client is listening for connection attempts. This state last indefinitely until the client reports some change. The natural progression from this state is for the client to report that a connection has been accepted or is in progress of being accepted, by calling self.connecting() or self.connected(). The client may also report that listening failed (e.g. accept() returned an unexpected error such as ENOMEM) by calling self.listen_error(), in which case the FSM will back off and eventually return ovs.reconnect.CONNECT from self.run() to tell the client to try listening again.rtN)rDrrrCrr4rNr r r listenings zReconnect.listeningcCs |jtjkr|||dSdS)aTell this FSM that the client's attempt to accept a connection failed (e.g. accept() returned an unexpected error such as ENOMEM). If the FSM is currently listening (self.listening() was called), it will back off and eventually return ovs.reconnect.CONNECT from self.run() to tell the client to try listening again. If there is an active connection, this will be delayed until that connection drops.N)rDrrrsrOrrpr r r listen_errors zReconnect.listen_errorcCs>|jjs|||d|j||tj||_dSdS)aTell this FSM that the connection was successful. The FSM will start the probe interval timer, which is reset by self.activity(). If the timer expires, a probe will be sent (by returning ovs.reconnect.PROBE from self.run(). If the timer expires again without being reset, the connection will be aborted (by returning ovs.reconnect.DISCONNECT from self.run().z %s: connectedN) rDrrurCrr4rr7rErNr r r connecteds   zReconnect.connectedcCs|||||dS)zjTell this FSM that the connection attempt failed. The FSM will back off and attempt to reconnect.N)rursrwr r rconnect_faileds zReconnect.connect_failedcCs$|jtjkr ||tj||_dS)zTell this FSM that some activity occurred on the connection. This resets the probe interval timer, so that the connection is known not to be idle.N)rDrr7r4r.rNr r ractivitys  zReconnect.activitycCr[)aTell 'fsm' that some attempt to receive data on the connection was made at 'now'. The FSM only allows probe interval timer to expire when some attempt to receive data on the connection was received after the time when it should have expired. This helps in the case where there's a long delay in the poll loop and then reconnect_run() executes before the code to try to receive anything from the remote runs. (To disable this feature, pass None for 'now'.)N)r/rNr r rreceive_attempteds zReconnect.receive_attemptedcCs|jtjkr|jd7_|tjkr|jd7_|jj}|j}||kr7|r0|j||j7_|j d7_ t d|j |j f||_||_ dS)Nr,z%s: entering %s)rDrr)rJr7rKrrLrErMr2r3rr)rOrrDconnected_before connected_nowr r rr4s   zReconnect._transitioncCs0|j||}|dur||kr|j||SdS)aAssesses whether any action should be taken on this FSM. The return value is one of: - None: The client need not take any action. - Active client, ovs.reconnect.CONNECT: The client should start a connection attempt and indicate this by calling self.connecting(). If the connection attempt has definitely succeeded, it should call self.connected(). If the connection attempt has definitely failed, it should call self.connect_failed(). The FSM is smart enough to back off correctly after successful connections that quickly abort, so it is OK to call self.connected() after a low-level successful connection (e.g. connect()) even if the connection might soon abort due to a failure at a high-level (e.g. SSL negotiation failure). - Passive client, ovs.reconnect.CONNECT: The client should try to listen for a connection, if it is not already listening. It should call self.listening() if successful, otherwise self.connecting() or reconnected_connect_failed() if the attempt is in progress or definitely failed, respectively. A listening passive client should constantly attempt to accept a new connection and report an accepted connection with self.connected(). - ovs.reconnect.DISCONNECT: The client should abort the current connection or connection attempt or listen attempt and call self.disconnected() or self.connect_failed() to indicate it. - ovs.reconnect.PROBE: The client should send some kind of request to the peer that will elicit a response, to ensure that the connection is indeed in working order. (This will only be returned if the "probe interval" is nonzero--see self.set_probe_interval()).N)rDrr)rOrrr r rr s'z Reconnect.runcCs0||}|dur|dkr||dSdSdS)z[Causes the next call to poller.block() to wake up when self.run() should be called.Nr)timeout timer_wait)rOpollerrrr r rwait8s zReconnect.waitcCs,|j||}|dur||}td|SdS)zReturns the number of milliseconds after which self.run() should be called if nothing else notable happens in the meantime, or None if this is currently unnecessary.Nr)rDrr&)rOrr remainingr r rr?s  zReconnect.timeoutcCs|jjS)zReturns True if this FSM is currently believed to be connected, that is, if self.connected() was called more recently than any call to self.connect_failed() or self.disconnected() or self.disable(), and False otherwise.)rDrrTr r rrJszReconnect.is_connectedcC|jr||jSdS)zwReturns the number of milliseconds since 'fsm' was last connected to its peer. Returns None if never connected.N)rErNr r rget_last_connect_elapsedQ z"Reconnect.get_last_connect_elapsedcCr)zReturns the number of milliseconds since 'fsm' was last disconnected from its peer. Returns None if never disconnected.N)rFrNr r rget_last_disconnect_elapsedYrz%Reconnect.get_last_disconnect_elapsedcCsGdddt}|}|j|_|j|_|j|_|j|_|j|_|j|_||_|||_ | ||_ |j |_ |rF|j ||7_ |j |_ |j|_|jj|_||j|_|S)Nc@s eZdZdS)z"Reconnect.get_stats..StatsN)rrrr r r rStatsbsr)objectrIrErFr.r rMrrmsec_since_connectrmsec_since_disconnectrLrJrKrDrr state_elapsed)rOrrstatsr r r get_statsas*     zReconnect.get_statscCs.|jdurdS|jdkr|jd8_dSdS)NTrr,Fr\rTr r r __may_retryxs  zReconnect.__may_retryN)/rrr__doc__rrrr#r)r7r5rrPrRrUrVrWrYrZr]r^r_r`rarbrdrerfrgrhrsrurvrxryrzr{r|r4rrrrrrrrcr r r rrsX     !      S   -  r) rmovs.utilovsovs.vlogr"r(r6utilror2Vlogrrr r r rs