RFC 793 why to send reset and when to send reset

  A variety of other cases are possible, all of which are accounted for
  by the following rules for RST generation and processing.

  Reset Generation

  As a general rule, reset (RST) must be sent whenever a segment arrives
  which apparently is not intended for the current connection.  A reset
  must not be sent if it is not clear that this is the case.

  There are three groups of states:

    1.  If the connection does not exist (CLOSED) then a reset is sent
    in response to any incoming segment except another reset.  In
    particular, SYNs addressed to a non-existent connection are rejected
    by this means.

    If the incoming segment has an ACK field, the reset takes its
    sequence number from the ACK field of the segment, otherwise the
    reset has sequence number zero and the ACK field is set to the sum
    of the sequence number and segment length of the incoming segment.
    The connection remains in the CLOSED state.

    2.  If the connection is in any non-synchronized state (LISTEN,
    SYN-SENT, SYN-RECEIVED), and the incoming segment acknowledges
    something not yet sent (the segment carries an unacceptable ACK), or
    if an incoming segment has a security level or compartment which
    does not exactly match the level and compartment requested for the
    connection, a reset is sent.

    If our SYN has not been acknowledged and the precedence level of the
    incoming segment is higher than the precedence level requested then
    either raise the local precedence level (if allowed by the user and
    the system) or send a reset; or if the precedence level of the
    incoming segment is lower than the precedence level requested then
    continue as if the precedence matched exactly (if the remote TCP
    cannot raise the precedence level to match ours this will be
    detected in the next segment it sends, and the connection will be
    terminated then).  If our SYN has been acknowledged (perhaps in this
    incoming segment) the precedence level of the incoming segment must
    match the local precedence level exactly, if it does not a reset
    must be sent.

    If the incoming segment has an ACK field, the reset takes its
    sequence number from the ACK field of the segment, otherwise the
    reset has sequence number zero and the ACK field is set to the sum
    of the sequence number and segment length of the incoming segment.
    The connection remains in the same state.



[Page 36]                                                               


September 1981                                                          
                                           Transmission Control Protocol
                                                Functional Specification



    3.  If the connection is in a synchronized state (ESTABLISHED,
    FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT),
    any unacceptable segment (out of window sequence number or
    unacceptible acknowledgment number) must elicit only an empty
    acknowledgment segment containing the current send-sequence number
    and an acknowledgment indicating the next sequence number expected
    to be received, and the connection remains in the same state.

    If an incoming segment has a security level, or compartment, or
    precedence which does not exactly match the level, and compartment,
    and precedence requested for the connection,a reset is sent and
    connection goes to the CLOSED state.  The reset takes its sequence
    number from the ACK field of the incoming segment.

  Reset Processing

  In all states except SYN-SENT, all reset (RST) segments are validated
  by checking their SEQ-fields.  A reset is valid if its sequence number
  is in the window.  In the SYN-SENT state (a RST received in response
  to an initial SYN), the RST is acceptable if the ACK field
  acknowledges the SYN.

  The receiver of a RST first validates it, then changes state.  If the
  receiver was in the LISTEN state, it ignores it.  If the receiver was
  in SYN-RECEIVED state and had previously been in the LISTEN state,
  then the receiver returns to the LISTEN state, otherwise the receiver
  aborts the connection and goes to the CLOSED state.  If the receiver
  was in any other state, it aborts the connection and advises the user