Your voltage setpoint should be fine. All it is used for is to make sure that the system is actively charging. It will then check the amps to determine when the batteries are full. Those amps are what the charged amps multiplier is. These are also the amps we call "return amps", "ending amps", "tail amps", "absorption transition amps" in other threads that you probably have seen. It will take the % of the battery capacity you have programmed in for your case. For AGM, the two percent is probably a bit high, especially when new. Lifeline says .5% for full, and most will go even lower than that. At 2% you will get a fully charged light before the batteries are totally full. Neither of these settings affect the AH counting, just the charged light. What will affect the AH readout is the charge efficiency, but only on recharge side. You want to get it to the point that you are around 100% full when the fully charged light comes on. It will vary with depth of discharge, etc, so not totally accurate on recharge. It does reset to 100% as soon as you start to discharge so the discharge will correct, if you got a full charge.
The wiring for the shunt looks OK from what is visible. All the grounds should be one side, and only the battery negative on the other. No other wires should go to the battery negative.
On the solar leak. Are you saying the Trimetric still reads, or does the coach power up, with the battery ground off? If the Trimetric stays on, there is a problem with wiring, most likely. If the solar comes on and powers the van, it probably is wired OK but running without battery reference, because the grounds are tied at the non battery side of the shunt. On some controllers this is bad thing to do, unless the panels are shut off. The problem comes if you are in sun, panels on, and you connect the solar to the coach without any batteries. You can get a big voltage spike that many harm electronics on the van 12v wiring.