Well, based on this response it's clear you've probably put more thought into this subject than I have, but I remain unconvinced.
I'll give you that the terms fairness and justice can, in many ways, be used interchangeably. I believe that the idea of "justice" is also something that is completely imaginary and therefore it is a concept I don't feel it is worth any time thinking about as an adult. There is no "justice" in this world. Bad things happen to good people and good things happen to bad people. If you spend your time worrying about "righting the wrongs" I feel you are going to be a very unhappy individual. Life is too short to spend time obsessing about what happens to other people. I choose to spend my life focused on living the best that I can. I do my best to be as ethical as I can be, and I teach my children that as well. There are others that do not live their lives that way, and sometimes their "unethical" behavior leads to great outcomes for them. That is something they have to live with, it is not something that concerns me.
I like the quote from Randy Newburg on this one, "You can have justice, or you can have peace, you can't have both". He says this in regards to having a happy marriage, but I think that advice is applicable to living life in general.
I enjoy these discussions, I gather from the above that you may not. If so, feel free to skip this.
I view it is like rights (which anly exist for sentient beings, or conversely are completely imaginary). I only believe negative rights are a valid concern for society, likewise I only believe negative justice/fairness is a valid concern.
I'll use the Second Amendment as an example. I have a right to keep and bare arms, in the negative since this means others should not inhibit me from doing so. The negative right only demands abstaining from action on the part of others (i.e. they cannot take my guns away). A positive right would mean someone (lets say the government just because) has an obligation to provide me with arms.
So, I would say people have an ethical obligation not to inflict injustice on others, however there is NO obligation to insure the lives of others are devoid of injustice.
Granted, the result of this up to this point is very similar in practice to what you describe.
Where I differ, I feel there is a vested interest in punishing intentional injustice. At the most basic level, I believe it is ethical to fund the police and judicial system even though doing so represents being concerned with Injust actions carried out by others.
It is only hard to live with having done something if one believes it is wrong. I am not content to say having "to live with" it is enough for murderers and rapists (again, to use the most extreme example) and I do concern myself with their actions even if those actions do not effect me or mine.
While I value peace, there are things I value significantly more. So, for myself, I prefer the quote by John Stuart Mill "War is an ugly thing, but not the ugliest of things. The decayed and degraded state of moral and patriotic feeling that thinks that nothing is worth war is much worse. The person who has nothing for which he is willing to fight, nothing which is more important than his own personal safety, is a miserable creature and has no chance of being free unless made and kept so by the exertions of better men than himself."
Note, I very specifically lack the ability to say that quote applies to you, so please don't take it as me trying to imply anything about your character. It is just the quote that popped into my head when I read your Newburg quote as in general justice (in the negative since) is something I value more than peace. I value it more than my marriage as well and if I became an abusive husband my wife would be completely justified in ending our marriage on the grounds that I was inflicting injustice on her, I would even go so far as to say she would be wrong to keep our daughters in such a situation.
Now, in regards to positive justice, which is what many people ruin marrages over, I'm in complete agreement with Newburg and value peace more. Same applies to society in general.