Multiple Internet connections

I have a Linux router/gateway/firewall that has two Internet
connections. The main connection is symmetrical (fast upload and
download), while the other connection is asymmetrical (fast download,
slow upload).

I've got some ideas so far - I'm hoping for some comments or hints to
tell me if I'm on the right path. I haven't tried implementing any of
this as yet.


Up until now I've just been using the main connection, with the the
second connection being used with different network equipment. But I'd
like to try to make use of both connections.

The ideal setup I would like is this:

$IF1 is the main interface for most traffic.

Outgoing http traffic should be split between $IF1 and $IF2.

If $IF1 goes down, all outgoing traffic should go through $IF2 (and
similarly if $IF2 goes down, everything should go through $IF1).

For any incoming traffic, replies should go back through the same
interface as the incoming packet.



The idea is the main downstream-heavy web traffic will benefit from the
extra bandwidth of the secondary connection, while things like email
will continue to use the symmetrical main connection. And in the event
of a failure on the main line, we will still have access.



As far as I can see, I could get a simple fail-over by just making two
default routes, one for each interface but with a higher metric for
$IF2. However, that would not get me any sort of load balancing and
replies to anything coming in on $IF2 would go out on $IF1.


I've been looking at
<http://lartc.org/howto/lartc.rpdb.m....html>.

What I need, I think, is to set up two routing tables (in
/etc/iproute2/rt_tables) T1 and T2, and put $IF1 and its default route
into T1, and similarly for $IF2 in T2:

ip route add $P1_NET dev $IF1 src $IP1 table T1
ip route add default via $P1 table T1
ip route add $P2_NET dev $IF2 src $IP2 table T2
ip route add default via $P2 table T2

The new tables can be added into the main routing by:

ip rule add from $IP1 table T1
ip rule add to $IP1 table T1
ip rule add from $IP2 table T2
ip rule add to $IP2 table T2


However, now I'm a bit stuck. From lartc.org and the man page for "ip",
I can see how to set up the routing so that it will work for fail-over:

ip route add default via $P1 metric 0
ip route add default via $P2 metric 10


lartc.org also gives an example of load balancing:

ip route add default scope global nexthop via $P1 dev $IF1 \
weight 1 nexthop via $P2 dev $IF2 weight 1

However, I only want such load balancing for http traffic - I certainly
don't want have my outgoing smtp traffic on the low upstream connection!

As a general idea, I think I am looking to use iptables rules to mark
packets, and then using those marks to select the routing table. I
think I then need another table for the balanced http routing. For example:

# For forwarded packets
iptables -A PREROUTING -t mangle -p tcp --dport 80 -j MARK \
# For packets from the firewall machine, for completeness
iptables -A OUTPUT -t mangle -p tcp --dport 80 -j MARK \

ip route add default via $P1 metric 0
ip route add default via $P2 metric 10

ip route add default scope global nexthop via $P1 dev $IF1 \
weight 1 nexthop via $P2 dev $IF2 weight 1 table balanced

ip rule add fwmark 1 table balanced


Any comments, corrections, hints, or links?

Thanks,

David