The Problem [1]:

The Problem [1]:
In em_process_receive_interrupts(), if em_get_buf() fails, which
may be a common case since it calls m_getcl() with MB_DONTWAIT,
then the rest of the RX ring are _not_ processed at all.  The RX
ring may contain many, or even worse is full of, ready RX descriptors.
RDT will not be updated too, in this case.  So hardware will take
this situation as there is _no_ free space in RX ring, and this
probably leads to following catastrophe:
too many RX engine overruns -> RX engine stalls -> interrupts stall [2]

The Fix:
If em_get_buf() fails then
- The current RX descriptor still will be reaped, but the frame,
  which belongs to the current RX descriptor, is not delivered to
  the upper layer.
- RX ring processing keeps going as if there is no failure happens.

Thank dillon@ to help diagnose the problem and give various hints [3]

Thank Mike Tancsa <mike@sentex.net> to provide enough information
to locate the problem.

Reviewed-by: dillon@
Reported-by: Mike Tancsa <mike@sentex.net>
Tested-by: Mike Tancsa <mike@sentex.net> (82571EB copper dual port)
           me (82540EM)

#
# [1] This problem unveiled itself when Mike tried to bench packet
#     forwarding performance.  The information he had provided that
#     is critical to locate the problem is:
#     1) Both polling(4) and normal mode (interrupt based) do not work.
#        This means that interrupt processing possibly not the root of
#        the problem (pointed out by dillon@)
#     2) One line in the output of sysctl hw.em0.debug_info=1:
#        ...
#        em0: Std mbuf cluster failed = 2
#        ...
#        This statistics is the value of adapter->mbuf_cluster_failed,
#        which is updated only when m_getcl() in em_get_buf() fails
#
# [2] This is just a guess from the output of 'vmstat -i' provided by Mike
#     Before his benching:
#     interrupt                   total       rate
#     ...
#     em0                            28          0
#     ...
#     After his benching (i.e. em0 choked):
#     interrupt                   total       rate
#     ...
#     em0                            62          0
#     ...
#     See, only 34 interrupts came o_O
#
# [3] There is still prossibility that RX engine gets confused by
#     the driver.  It is described by dillon@:
#     (I didn't find the URL in the archive, so it is pasted here)
#     "...
#      Clearly when that case occurs ALL the receive frames will be full.
#      Lets look at a degenerate case:
#
#      [0 ...................... N-1]
#      * RDH set to 0
#      * RDT set to N-1
#      * N frames come in  RDH is set to N-1 (??)
#      * We process N frames
#      * The frame at RING[N-1] is cleaned up
#      * i = N
#      * We set RDT to i-1 == N-1.  It's the same value it was set to before
#        we processed all N frames.  The receive engine will think that the
#        ring is still full when it is empty.
#
#      I Think what we need to do here is set RDT to N-2 (mod N of course) in
#      the case where we have processed ALL N frames.  I'll bet the firmware
#      is getting confused in the overrun case because we are setting RDT to
#      the same value it was set at before.
#      ..."
#